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3  Volume  I  of  the  Operational  Suitability  Guide  was  prepared  by  the 
Office  of  the  Director  of  Operational  Test  and  Evaluation 
■CDOTSE)  to  provide  an  overview  of  those  issues  that  are  included 
the  general  subject  of  operational  suitability,  and  to  provide 
background  information  for  DOTSE  Staff  Assistants  to  use  when 
examining  operational  suitability  subjects.  The  document 
discusses  each  of  the  operational  suitability  elements,  i.e., 
reliability,  maintainability,  safety,  human  factors, 
availability,  logistic  supportability,  etc.;  the  parameters  used 
during  OTSE  for  each  element;  and  key  points  to  be  remembered  in 
the  area  of  each  element.  The  report  includes  as  an  appendix  the 
Service  Operational  Test  Agencies  (OTA)  Memorandum  of  Agreement 
on  Common  Reliability,  Availability,  and  Maintainability 
Terminology.  ^ - 
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FOREWORD 


The  effective  operational  test  and  evaluation  (OT&E)  of  defense  systems  is  a  critical  part  of  the 
long-term  program  to  provide  for  the  proper  defense  for  the  United  States.  The  Department  of 
Defense  has  an  established  process  for  planning  and  conducting  operational  tests  and  for 
evaluating  the  data  that  result  from  those  tests. 

Volume  I  of  this  Operational  Suitability  Guide  was  prepared  by  the  Office  of  the  Director  of  Op¬ 
erational  Test  and  Evaluation  (DOT&E)  to  provide  an  overview  of  those  issues  that  are  included 
in  the  general  subject  of  operational  suitability,  and  to  provide  background  information  for 
DOT&E  Staff  Assistants  to  use  when  examining  operational  suitability  subjects.  This  volume 
serves  as  a  tutorial  for  OT&E  personnel. 

Volume  n,  OS  Source  Documentation:  Supporting  References,  focuses  on  the  review  of  specific 
OT&E  documents  and  coverage  of  operational  suitability  in  those  documents.  The  information 
in  Volume  II  is  intended  to  supplement  the  policy  and  procedures  contained  in  DoD  directives 
and  manuals.  This  document  does  not  establish  new  requirements  for  operational  test  and 
evaluation  documentation. 

Volume  I  is  organized  in  the  following  manner.  Chapter  1  discusses  operational  suitability  in 
general,  and  includes  a  detailed  discussion  on  each  of  the  elements  that  are  listed  in  the  definition 
of  suitability.  Chapter  2  discusses  additional  issue  areas  that  deserve  emphasis.  Chapter  3 
contains  Annex  A  to  the  Operational  Test  and  Evaluation  Agencies  (OTAs)  Memorandum  of 
Agreement  (MOA)  on  Multi-Service  OT&E.  This  annex  comprises  a  listing  of  "Common 
Reliability,  Availability,  and  Maintainability  (RAM)  Terminology,”  including  common  RAM 
parameters  and  a  listing  with  definitions  of  die  other  RAM  terms  used  by  the  Services. 

If  questions  or  comments  arise  while  reviewing  or  using  this  guide,  they  should  be  forwarded  to 
the  primary  author 


Dr.  Elizabeth  Rodriguez 
The  Office  of  the  Director  of  Operational  Test 
and  Evaluation 

Office  of  the  Secretary  of  Defense 
The  Pentagon,  Room  1C730 
Washington,  DC  20301 
(202)  697-3895 
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Chapter  1 

OPERATIONAL  SUITABILITY 


INTRODUCTION 

Operational  suitability  is  defined  as 

the  degree  to  which  a  system  can  be  placed  satisfactorily  In  field  use 
with  consideration  given  to  availability,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety, 
human  factors,  manpower  supportability,  logistics  supportablllty,  docu¬ 
mentation,  and  training  requirements. 

The  suitability  elements  listed  in  this  definition  ate  discussed  on  the  following  pages.  However, 
the  definition  lists  only  some  of  the  items  that  may  be  suitability  issues  for  a  particular  system; 
additional  issues  may  be  dictated  by  the  system's  mission(s)  or  the  planned  logistics  support 
process  that,  during  operational  test  and  evaluation  (OT&E)  planning,  may  need  to  be 
considered.  Chapter  2  discusses  five  of  these  additional  issues;  suitability  modeling  and 
simulation,  integrated  diagnostics,  environmental  factors,  electromagnetic  environmental  effects 
(E3),  and  software  suitability. 

Chapter  1  and  2  are  organized  to  present,  as  sub-sections,  a  definition  of  each  suitability  element 
or  issue  to  be  discussed,  some  of  die  parameters  that  apply,  and  key  points  that  must  be 
considered  in  its  application  to  operational  suitability.  Key  points  and  milestone  activities  that 
serve  to  support  the  overall  operational  suitability  aspect  of  OT&E  are  set  forth  below. 

KcY  POINTS 

Effective  systems  must  be  snitat  le. 

Poor  suitability  may  preclude  the  use  of  an  otherwise  effective  system  in  combat.  Any  limitation 
that  suitability  imposes  on  die  effective  employment  of  a  weapon  system  must  be  identified  and 
evaluated.  The  suitability  portion  of  OT&E  must  be  planned  and  conducted  to  determine  what, 
if  any,  limitation  is  likely  to  exist  when  the  system  is  placed  in  operation. 


Suitability  issues  that  have  the  highest  risk  siusf  be  identified. 

While  all  suitability  issues  must  be  satisfactory,  the  risks  associated  with  these  issues  vary.  The 
array  of  suitability  topics  for  an  individual  system  usually  involves  a  number  of  suitability  criti¬ 
cal  operational  issues  (COls).  Identifying  these  critical  issues  allows  focus  and  attention  to  be 
directed  to  those  areas  in  need  of  detailed  and  careful  examination.  Developing  this  focus  is  an 
essential  pan  of  die  early  Service  planning  of  every  OT&E.  Independent  examination  of  the 
Systran  description,  mission  description,  and  planned  support  concept  by  die  DGT&E  staff 
provides  the  basis  for  evaluating  Service-identified  critical  suitability  issues. 
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KEY  POINTS  (Corit’d) 

The  operating  scenario  drives  the  suitability  demands. 

The  demands  for  maintenance  and  supply  support  depend  on  the  intensity  of  the  system's  us:. 
Operating  hours  per  day,  miles  per  day,  flying  hours  per  day,  that  ponton  of  the  operation 
conducted  at  high  speed  conditions,  any  ratio  of  different  mission  types,  etc.,  all  are  important 
factors  for  estimating  the  intensity  of  operational  use.  These  factors  must  be  estimated  for  the 
planned  operational  scenario  and  then  considered  as  part  of  the  Services’  planning  for  the 
operational  testing.  The  Services  specify  these  mission  parameters  and  scenarios  in  various 
documents,  e.g.,  Operational  Mode  Summary.  The  way  in  which  the  system  is  used  during 
operational  testing,  including  the  realism  of  the  mission  scenariofs)  and  the  environmental 
conditions,  is  critical  to  constructing  a  test  program  that  provides  realistic  operational  suitability 
demonstrations  and  produces  realistic  suitability  test  data. 

Terminology  needs  to  be  consistent 

Table  1-1  presents  a  hierarchy  of  terminology  that  relates  the  conduct  of  OT&E,  Le., 
characteristic,  parameter,  threshold.  The  evaluation  of  operational  suitability  involves  selecting 
and  examining  characteristics  that  relate  to  each  of  the  suitability  issues.  The  characteristics  for 
each  of  the  issues  are  selected,  considering  die  system's  operating  and  support  scenarios  and  the 
relative  criticality  of  the  areas  included  in  each  of  the  issues.  As  put  of  the  planning  for 
operational  test,  each  characteristic  will  have  one  or  more  parameters  identified.  To  evaluate  the 
data  that  result  from  the  operational  tests  in  each  of  these  areas,  the  parameter  measurements 
must  be  compared  to  thresholds  that  were  previously  established.  The  table  includes  an  example 
of  how  these  terms  might  be  applied  when  reliability  is  an  operational  suitability  issue. 

Table  1-1  Hierarchy  of  Terminology 


TERM 

EXAMPLE 

Characteristic 

Mission  Reliability 

Parameter 

Mean  Time  Between  Operational 

Mission  Failure  (MTBOMF) 

Threshold 

MT80MF  Shall  Be  at  Least  300  Hours 

There  are  always  limitations  to  operational  testing. 

Resource  and  safety  constraints  often  impose  limitations  on  the  conduct  of  the  testing.  The 
number  of  test  articles  available,  the  number  of  test  hours,  the  availability  of  all  support  systems, 
and  the  realism  of  the  logistics  system  almost  always  are  limited.  The  importance  of  these 
limitations  and  their  effect  on  the  suitability  results  must  be  addressed  in  the  Test  and  Evaluation 
Master  Plan  (TEMP),  test  plan,  and  test  report. 
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Operational  suitability  applies  to  each  level  of  support 


The  Services  use  various  support  structures  (i.e.,  levels  of  support)  for  weapons  systems.  Table 
1-2  presents  some  examples  of  the  levels  of  support  that  may  apply.  For  most  systems,  the  suit¬ 
ability  elements  (e.g.,  maintainability,  training,  documentation)  apply  differently  to  each  support 
level.  While  only  the  first  and  second  levels  of  support  may  be  available  at  the  operational  test 
site,  the  evaluation  should  consider,  if  possible,  all  applicable  support  levels. 


Table  1-2  Variance  in  the  Definitions  of  Support  Levels 


KEY  POINTS  (Cont’d) 

Operational  suitability  has  many  dimensions. 

It  is  impossible  to  combine  the  many  quantitative  and  qualitative  aspects  of  suitability  into  a 
single  measure  —  the  unique  aspects  of  each  system  impose  a  different  priority  on  the  suitability 
issues.  The  overall  assessment  of  a  system's  suitability  is  an  expert  judgment  based  upon  a 
multitude  of  factors.  The  suitability  evaluation  must  assess  what  the  OT  results  say  about  the 
likelihood  that  the  system  can  be  satisfactorily  placed  in  field  use  in  the  intended  operating 
environment  If  an  area  of  suitability  is  less  than  the  level  stated  in  the  requirements,  the 
evaluation  should  estimate  the  impact  of  this  deficiency  on  the  system. 

MILESTONE  ACTIVITIES 

During  the  early  system  definition  studies  and  analyses,  the  critical  operational  issues  (COls)  in 
the  suitability  area  should  be  identified  The  initial  Test  and  Evaluation  Master  Plan  (TEMP) 
should  discuss  these  issues.  The  characteristics  that  relate  to  COls  should  be  identified  by 
Milestone  I,  the  Concept  Demonstration/Validation  Decision.  The  system  mission  profiled)  and 
life  profile  also  should  be  defined  by  Milestone  I  and  documented  in  the  Operational  Mode 
Summary,  or  similar  document.  (The  life  {nofile  is  a  time-phased  description  of  the  events  and 
environments  that  an  item  experiences  from  manufacture  to  final  expenditure  of  the  item  or  its 
removal  from  the  operational  inventory.  It  includes  one  or  more  mission  profiles,  in  addition  to 
any  storage,  transportation,  maintenance,  or  exercise  events  and  environments  that  the  item  will 
experience.)  The  early  definition  of  these  profiles  does  not  imply  that  the  profiles  are  "in 
concrete"  and  will  not  be  revised. 

By  Milestone  II,  the  Full-Scale  Development  Decision,  the  program  manager  and  the 
developing  contractors  should  have  a  reasonably  well  defined  "system-level"  design.  The 
required  ievel  of  reliability  and  maintainability  should  be  known.  The  maintenance  diagnostics 
approach,  the  maintenance  concept,  and  the  general  level  of  support  requirements  should  be 
established.  The  training  concept  should  be  understood.  The  relationship  between  the  system's 
reliability  and  maintainability  requirements  and  the  maintenance  concept  should  be  defined  and 
in  balance  with  the  planned  logistics  support  concept  High  reliability  systems  can  be  supported 
with  unique  logistics  support  systems,  e.g.,  missiles  that  are  handled  as  "wooden  rounds."  (A 
“wooden  round”  is  a  missile  or  munition  that  is  handled  in  the  operating  unit  as  a  single  assem¬ 
bly.  There  is  no  plan  or  capability  to  isolate  faults  or  to  disassemble  the  item  at  the  operating 
unit)  Weaknesses  in  these  areas  or  lack  of  detailed  knowledge  may  cause  problems  as  the 
program  proceeds.  Lack  of  definition  at  Milestone  II  also  may  result  in  the  developing 
organizations  or  contractors  having  differing  views  of  some  aspects  of  the  program.  This  can 
lead  to  inconsistency  between  the  support  planning  and  the  system's  detailed  design.  The  defini¬ 
tion  of  the  system  and  its  support  concept  are  needed  to  define  the  OT&E  criteria  and  produce 
the  Milestone  II  TEMP. 

If  there  is  a  Milestone  IRA  (the  Low  Rate  Initial  Production  (LRIP)  decision),  an  operational 
assessment  report  or  the  first  of  the  OT&E  reports  should  show  the  status  of  the  system  in 
meeting  its  operational  suitability  requirements  and  satisfactorily  resolving  the  suitability  issues. 
Testing  should  be  complete  in  some  operational  suitability  areas,  and  results  compared  to  the 
criteria  and  the  evaluation  results  reported  during  the  Defense  Acquisition  Board  (DAB)  process. 
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The  CT&E  report  that  supports  Milestone  KB  (the  Full  Rate  Production  Decision)  should 
update  the  previous  information  and  (if  not  complete)  should  have  an  expanded  evaluation  of 
operational  suitability  subjects.  The  test  and  evaluation  report  should  compare  the  results  to  the 
threshold,  highlight  the  current  “status”  of  the  system,  and  describe  areas  that  have  changed 
status,  i.e.,  from  "deficient"  or  "unsatisfactory"  to  "satisfactory."  This  report  should  contain  the 
final  assessment  of  the  question,  'Is  the  system  suitable?” 


SH  AVAflABlU 


Availability  is  defined  as 

a  measure  of  the  degree  to  which  an  Item  Is  In  an  operable  and 
committaUe  state  at  tfie  start  of  a  mission  when  the  mission  Is  called  for 
at  an  unknown  (random)  time. 

This  definition  of  availability  addresses  systems  that  spend  a  portion  of  their  time  in  a  "ready" 
status,  and  at  some  undetermined  time  are  required  to  initiate  a  mission.  The  discussion  of  a 
system’s  availability  must  consider  the  type  of  system  being  considered.  Items  being 
operationally  tested  range  from  entire  aircraft,  to  complex  ship  combat  systems,  to  relatively 
small  man-portable  systems  and  items  that  are  only  part  of  a  complex  combat  system.  Some  sys¬ 
tems  spend  most  of  their  time  in  a  readiness  status,  always  available  to  perform  a  single  mission 
(e.g.,  a  strategic  missile  system).  Other  types  of  systems  with  other  operating  scenarios  require 
other  measures.  Some  are  "continuous-use"  systems  that  are  required  to  perform  twenty-four 
hours  a  day  (e.g.,  command  and  control  computer  systems,  communications  systems,  or  warning 
systems).  Other  systems  spend  time  in  a  ready  status  and  perform  repetitive  missions  when 
called  upon  (e.g.,  tactical  aircraft).  In  many  cases,  the  call  to  perform  a  mission  is  not 
necessarily  random;  the  operational  commander  e^rtcises  some  control  over  when  the  particular 
system  is  required  to  perform  a  mission.  Because  of  the  degree  of  control  over  the  scheduling  of 
tactical  aircraft  sorties,  some  aircraft  systems  are  better  characterized  by  using  "sorties  per 
aircraft  per  day,"  or  "sortie  rate,"  as  a  measure  of  bow  available  die  system  is  to  perform  its 
mission.  The  probability  of  being  available  for  the  first  mission  may  be  very  different  than  the 
probability  of  heing  available  for  second  and  later  missions.  Examination  of  the  availability  far 
each  of  these  cases  requires  consideration  of  die  different  perspective  in  each  case.  In  some 
cases,  manpower  levels  may  limit  availability. 


PARAMETERS 

The  multi-Service  memorandum  of  agreement  (MOA)  (Chapter  3)  has  two  definitions  for  opera¬ 
tional  availability,  AQ.  The  first  is 


_ Total  Uptime _ 

Total  Uptime  +  Total  Downtime 


when  operated  in  an  operational  mission  scenario.  The  second  is 

^  _  Number  of  systems  ready 

0  Number  of  systems  possessed 


The  first  equation  can  be  used  during  the  OT  of  subsystems,  and  in  situations  where  it  is  possible 
for  the  system  to  be  in  a  state  other  than  "up"  or  "down."  An  example  would  be  a  situation 
where  there  is  an  interruption  of  die  testing  for  redeployment  of  the  test  forces,  system 
reconfiguration,  or  other  activity.  The  operational  test  plan  must  state  clearly  how  these  periods 
of  "no  test"  are  defined  and  who  will  determine  when  these  periods  start  and  stop. 
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The  Services  use  other  methods  of  calculating  the  ratio  of  availability.  For  some  Army  and  Ma¬ 
rine  Corps  systems,  the  AQ  is  calculated  by  an  expanded  equation 


^ _ Operating  Time  +  Standby  Time _ 

°  Operating  Time  +  Standby  Time  +  Total  Corrective  Maint  Time 
+  Total  Preventive  Maint  Time  +  Total  Administrative  and 
Logistics  Downtime 

At  ither  times,  the  Services  may  use  parameters  such  as  the  percent  of  time  that  the  system  is 
Mission-Capable  (MC),  Full  Mission-Capable  (FMQ,  and  Partial  Mission-Capable  (PMC)  as 
measures  for  availability.  These  measures  are  dependent  on  the  list  of  system  equipment  that  is 
essential  for  each  system's  missions.  (A  full  listing  of  mission-essential  equipment  should  be 
contained  in  the  TEMP,  car  included  by  reference.)  By  definition, 

a  system  is  Tull  Mission-Capable"  when  it  has  all  mission-essential 
equipment  available  and  can  perform  any  of  its  missions: 

a  system  is  "Partial  Mission-Capable"  when  only  a  portion  of  the 
mission-essential  items  are  available,  but  can  perform  at  least  one,  but 
not  all,  of  its  missions; 

a  system  is  "Mission-Capable"  when  it  is  in  either  a  PMC  or  FMC 
condition; 

"Not  Mission-Capable"  means  that  the  system  does  not  have  the 
equipment  available  to  perform  any  of  its  missions. 

One  of  the  advantages  of  these  parameters  is  that  they  may  allow  OT&E  results  to  be  put  into 
terms  that  are  familiar  to  operational  commanders. 

Another  availability  measure  is  achieved  availability.  This  parameter  may  be  used  in  situations 
where  the  test  is  limited  in  the  logistics  area.  It  is  very  costly  to  procure  spares  or  to  have  a  rep¬ 
resentative  logistics  system  for  engineering  development  models  (EDM)  or  for  situations  where 
two  or  more  contractors  are  competing  by  producing  competitive  systems  that  are  to  be  evaluated 
in  an  operational  test  When  the  supply  support  is  limited  and  nomepiesentative,  the  total  ad¬ 
ministrative  and  logistic  downtime  component  of  operational  availability  cannot  be  evaluated. 
Achieved  availability  does  not  consider  the  downtime  associated  with  logistic  or  administrative 
delays.  The  equation  for  achieved  availability  is 

Achieved  _ _ Operating  Tune _ 

Availability  Operating  Time  +  Total  Corrective  Maintenance  Time 
+ Total  Preventive  Maintenance  Time 


Another  parameter  for  availability  relates  to  die  percentage  of  time  that  an  item  is  able  to  satisfy 
a  demand  for  its  service.  This  parameter  may  be  applied  to  systems  that  are  drawn  from  storage 
or  from  a  stockpile,  or  to  systems  that  are  maintained  in  a  standby  state  and  then  called  on  to 
support  a  mission  at  a  specific  time.  Demand  availability  is  usually  expressed  as  a  percentage. 


Demand  Availability  = 


Number  of  times  available 
Number  of  times  requested 


Additional  parameters  for  availability  are  listed  in  the  muld-Service  MOA  (See  Chapter  3). 
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Availability  is  a  critical  characteristic  that  should  be  discussed  in  the  early  planning 
documents. 

The  early  system  planning  documents  should  provide  a  basis  for  relating  availability  to  other 
system  characteristics  as  the  program  proceeds  through  later  acquisition  phases.  The  system 
requirements  should  identify  which  availability  parameter  is  most  meaningful  for  the  system. 
How  this  parameter  relates  to  the  operating  scenario  and  the  reliability  and  maintainability 
parameters  needs  to  be  understood. 

System  availability  is  difficult  to  measure  during  short  operational  testing  periods. 

During  operational  testing,  the  measured  value  for  availability  can  be  totally  unrepresentative  of 
what  might  be  expected  in  operational  service.  For  example,  in  a  short  test  period,  only  a  few 
failures  may  occur.  As  this  may  not  be  a  representative  number  of  failures,  the  resulting 
calculated  availability  may  be  very  optimistic.  For  an  immature  system,  the  time  to  identify 
problems  and  restore  the  system  to  an  operational  status  may  be  extremely  lengthy.  In  these 
cases,  the  limitations  on  the  availability  measure  must  be  recognized.  In  addition,  the  planned 
maintenance  and  supply  systems  may  not  be  in  place  because  the  system  is  not  yet  fielded  and 
may  not  be  fielded.  Modeling  and  simulation  may  be  useful  in  assessing  the  availability  in  these 
simtions. 

The  OT  planning  should  address  the  methods  of  measuring  times  for  the  availability 
evaluation. 

The  way  in  which  a  system  is  to  be  used  in  operation  will  determine  the  most  appropriate 
availability  parameter.  How  that  parameter  is  applied  to  the  system  in  question  will  help 
establish  how  the  system’s  operational  time  is  recorded  and  evaluated.  If  the  system  has  standby 
time,  or  time  that  will  not  be  included  in  the  availability  calculation,  than  the  test  planning  should 
address  the  specifics  of  how  these  items  will  be  handled.  If  the  OT  is  to  include  periods  during 
which  the  measure  of  time  is  not  to  be  included  in  die  availability  evaluation,  then  the  test  plan 
should  address  the  definition  and  likelihood  of  "no  test"  time,  and  indicate  how  this  time  will  be 
addressed  in  calculating  operational  availability. 

System  standby  time  may  be  important 

If  system  standby  time  is  included  in  the  calculation  of  availability,  then  the  ratio  of  die  standby 
time  to  active  system  operating  time  should  be  assessed  for  “reasonableness.’'  If  an  unreason¬ 
ably  high  ratio  of  standby  time  to  system  operating  time  is  evident  in  the  test  then  the  calculated 
operational  availability  will  be  unrealistically  high.  An  estimate  of  die  ratio  drat  will  be  observed 
in  combat  operations  should  be  described  in  the  Operational  Mode  Summary,  or  similar 
documents.  The  planning  for  the  operational  testing  should  address  what  ratio  is  planned  for  the 
testing  period,  and  how  this  compares  to  the  estimated  ratio  for  operational  use. 

Logistics  support  realism  should  be  an  objective  in  planning  for  operational  testing. 

To  provide  useful  insight  into  the  operational  availability  of  the  system  being  tested,  die  logistics 
support  being  provided  during  the  testing  should  be  as  realistic  as  possible.  Any  limitations  that 
exist  need  to  be  identified  prior  to  the  test,  and  included  in  the  evaluation  of  test  results. 
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Probability  of  completing  a  missioo  of 
Mission  Reliability  =  X  hours  without  a  critical  failure, 

tinder  specified  mission  conditions 


Probability  of  Success  =  Number  of  successful  attempts 

Total  number  of  attempts  _ ,  , 

Mission  reliability  also  can  be  staled  as  Mean  Time  (Miles,  Rounds,  etc.)  Between  Operational 
Mission  Failure  (MTBOMF).  This  parameter  can  be  used  for  continuously  operating  systems, 
such  as  communications  systems,  or  for  vehicles  or  artilleiy. 

Mean  Time  Between 
Operational  Mission 
Failure 

Another  parameter  that  sometimes  is  used  in  place  of  MTBOMF  is  Mean  Time  Between 
Mission-Critical  Failures  (MTBMCF),  which  has  a  similar  equation,  substituting  the  term 
"mission-critical  failures"  for  "operational  mission  failures."  In  both  cases,  the  definition  of  what 
constitutes  a  “mission  failure”  must  be  clearly  documented  for  the  applicable  system. 

The  parameter  "Mean  Time  Between  Failure”  (MTBF)  is  being  used  less  frequently  during 
operational  test  and  evaluation.  One  reason  for  this  is  the  confusion  that  may  exist  between  the 
use  of  MTBF  as  a  technical  parameter  in  contract  documents  or  in  DT&E,  and  the  use  of  a 
similar  parameter  as  an  OT&E  parameter.  (The  trend  in  the  Air  Force,  has  been  to  reserve  MTBF 
for  DT&E  and  contract  purposes,  and  to  use  other  parameters  with  operationally  oriented 
definitions  in  OT&E.)  If  MTBF  is  stated  in  a  TEMP  or  OT&E  plan,  the  definition  of  what  is 
considered  a  failure  must  be  included  (or  documented  in  a  reference)  in  sufficient  detail  to  ensure 
that  it  includes  all  operational  influences,  not  just  system  or  component  design  problems. 

Logistics  support  frequency  is  measured  as  die  time  between  events  requiring  inwhetiiilrti 
maintenance,  unscheduled  removals,  or  unscheduled  demands  for  spare  parts.  These  events  are 
considered  whether  or  not  mission  capability  is  affected.  Logistics  support  frequency  can  be 
expressed  as  Mean  Time  Between  Unscheduled  Maintenance  (MTBUM). 


Total  operating  time  (e.g.,  driving  time, 

. _ flying  tune,  or  system-on  time) _ 

Total  number  of  operational  mission  failures 


Mean  Time  Between 
Unscheduled 
Maintenance 


Total  number  of  incidents  requiring 
unscheduled  maintenance 


If  MTBF  is  used  as  a  measure  of  logistics  support  frequency,  the  definition  must  include  any 
appropriate  maintenance  events  that  are  not  the  result  of  failures  such  as  preventive  maintenance 
actions,  inspections,  calibrations,  or  no-fault-found  actions.  If  these  non-failure-caused 
maintenance  actions  are  not  included  in  the  calculation  of  MTBF,  then  die  MTBF  may  be 
significantly  higher  than  a  measurement  of  the  MTBUM  and  is  not  a  measure  of  logistics  support 
frequency.  Other  logistics  support  planning  such  as  manpower,  spares,  or  tire  amount  of  test 
equipment  will  be  understated  if  MTBF  is  assumed  to  be  the  parameter  that  determines  the  need 
for  these  resources. 

Another  logistics  support  parameter  that  is  indirectly  a  measure  of  reliability  is  Mean  Time 
Between  Removal  (MTBR).  This  parameter  is  used  to  measure  tire  frequency  of  failures  or 
maintenance  actions  that  require  some  item  of  equipment  to  be  removed  from  die  end  item. 
MTBR  indicates  the  demands  on  the  supply  system  for  replacement  items  and  for  repair 
processing  at  the  intermediate  or  direct  levels  of  maintenance. 
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Reliability  parameters  should  be  defined  early  in  a  program. 

By  Milestone  I,  the  critical  reliability  parameters  should  be  set  forth  by  the  user  or  user 
representatives  in  the  early  planning  documents.  These  descriptions  are  the  basis  for  considering 
reliability  and  in  relating  it  to  other  system  parameters. 

The  system's  operating  modes  can  drive  reliability. 

System  operating  modes  must  be  considered  in  determining  the  relative  importance  of  mission 
and  logistics  reliability.  Defense  systems  may  be  continuously  operating,  perform  repetitive 
missions,  or  be  one-shot  items.  The  reliability  parameter  must  be  selected  according  to  how  the 
system  is  employed.  Systems  that  are  continuously  operating  may  be  measured  by  Mean  Time 
Between  Operational  Mission  Failures  (MTBOMF).  Systems  dial  perform  repetitive  missions 
may  also  use  this  measure,  or  die  probability  of  completing  a  mission  without  a  mission  failtne. 

Firm  reliability  requirements  are  essential. 

Firm  requirements  must  be  established  in  the  Service  acquisition  documentation  for  each  of  the 
applicable  reliability  parameters  before  Milestone  H.  The  requirements  have  meaning  only  if 
they  include  a  detailed,  comprehensive  failure  definition  that  is  consistent  with  what  one  would 
expect  from  an  operational  user.  If  there  is  an  unusual  exception  (e.g.,  failures  caused  by  crew 
error  are  not  included),  this  exception  must  be  clearly  identified  in  the  TEMP  and  the  OT  plans. 

Reliability  measurements  can  require  lengthy  test  period*. 

High  reliability  systems  with  lengthy  times  between  failures  require  lengthy  test  times  in  order  to 
gather  sufficient  test  experience  to  produce  meaningful  measures  of  the  system's  reliability.  If 
extended  periods  of  test  time  are  not  achievable,  then  the  OT&E  must  be  structured  to  use  other 
methods  or  sources  of  data  to  evaluate  reliability.  Modeling  and  simulation  might  be  used  to 
focus  the  operational  testing  on  subsystems  of  greatest  risk  or  criticality.  The  data  of  technical 
testing  might  be  used  to  supplement  data  that  are  obtained  during  OT.  In  some  instances,  the 
only  solution  is  to  be  satisfied  with  reduced  confidence  levels,  since  the  cost  of  the  test  systems 
(e.g.,  complex  missile  systems)  is  too  high  to  permit  additional  test  articles  to  be  expended. 

Assumptions  are  made  in  reliability  test  planning. 

Long  test  periods  are  needed  to  accurately  measure  reliability  of  high  reliability  systems. 
Sometimes  a  greater  number  of  items  are  tested  for  a  short  time,  instead  of  a  few  items  for  a 
longer  time.  For  example,  a  good  test  program  for  a  system  with  a  MTBOMF  of  1000  hours 
might  be  to  test  some  number  of  systems  (e.g.,  four)  for  1000  hours  each.  Because  foe  long  test 
period  needed  to  obtain  1000  operating  hours  is  not  possible,  foe  test  is  planned  for  twenty 
systems  with  200  hours  each.  Although  this  will  result  in  foe  same  number  of  total  operating 
hours,  there  are  inherent  risks.  The  DOT&E  evaluator  will  need  to  determine  foe  likelihood  of 
significant  "wear-out”  failure  modes  -  are  there  significant  failure  modes  that  will  not  be  seen 
until  after  the  200  proposed  hours  of  testing?  Assessing  this  risk  requires  an  examination  of  the 
system.  The  responsible  Service  should  demonstrate  that  the  risk  is  acceptable  by  presenting 
other  test  data  that  demonstrate  that  significant  "wear-out"  failure  modes  have  not  occurred  in 
longer  duration  testing,  and  are  unlikely  to  occur  in  operation.  If  this  cannot  be  demonstrated. 
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then  alternative  test  approaches  should  be  considered,  e.g.,  testing  a  few  items  for  a  longer 
period,  having  test  units  tested  for  a  longer  period  in  other  test  phases,  etc. 

Early  OT  may  give  the  first  realistic  view  of  system  reliability. 

Initial  operational  testing  can  provide  insight  into  die  reliability  potential  of  the  system.  While  it 
is  difficult  and  time  consuming  to  verify  that  a  high  level  of  reliability  exists,  the  discovery  of 
reliability  deficiencies  is  not  difficult  if  the  system  is  truly  deficient  An  operational  assessment 
may  identify  areas  that  potentially  could  become  major  deficiencies.  A  critical  error  at  this  stage 
is  committed  when  the  reliability  data  that  are  the  result  of  early  testing  are  not  accepted  —  fail¬ 
ures  ate  “explained  away”  and  managers’  projections  that  accept  high  reliability  turn  out  to  be 
too  optimistic.  On  the  other  hand,  the  reliability  and  failure  experience  early  in  a  development 
program  may  not  be  representative  of  the  production  configuration;  the  results  indicated  by  early 
Developmental  Test  (DT)  data  may  change  significandy  by  the  time  the  system  reaches  maturity. 

If  there  are  any  deviations  from  the  planned  production  configuration,  they  should  be 
understood. 

To  evaluate  the  reliability  of  a  system  in  its  intended  operational  environment  requires  that  the 
items  being  tested  have  a  reasonable  degree  of  agreement  with  the  planned  production 
configuration.  All  significant  deviations  from  the  production  configuration  should  be  identified. 

Reliability  measures  can  have  statistical  confidence  calculations. 

One  measure  of  the  sufficiency  of  the  test  data  is  the  use  of  statistical  confidence  calculations. 
During  the  test  planning,  confidence  calculations  can  be  used  to  determine  how  much  testing  is 
needed  to  yield  a  certain  level  of  confidence  in  the  results.  After  the  test  data  have  been 
collected,  confidence  calculations  can  be  used  to  indicate  how  adequate  the  data  were  in 
determining  the  values  for  the  reliability  parameters.  A  good  reference  for  a  discussion  of 
confidence  levels  and  intervals,  and  how  they  apply  to  the  test  and  evaluation  environment,  is 
DoD  3235. 1-H,  "Test  and  Evaluation  of  System  Reliability,  Availability,  and  Maintainability  -  A 
Primer."  It  provides  a  reference  for  the  mathematical  aspects  of  RAM  and  discusses  confidence 
as  it  applies  to  planning  test  programs  and  evaluating  test  data.  The  discussion  is  based  on  tech¬ 
nical  R&M  characteristics  and  may  require  tailoring  when  applied  to  OT&E. 

Software  reliability  is  always  an  issue. 

As  defense  systems  become  more  and  more  software-intensive,  understanding  software's 
contribution  to  system  reliability  increases  in  importance.  Software  faults  and  errors  can  be  a 
major  problem  during  the  initial  phase  of  a  system's  operation,  and  they  should  be  viewed  within 
the  context  of  what  effect  die  faults  will  have  on  die  system's  performance.  Serious  faults  can 
result  in  mission  failures  and  should  be  treated  accordingly.  Easily  corrected,  minor  interrupts 
that  cause  no  system-mission  impact  may  be  significant  only  if  the  quantity  is  high  enough  for 
the  problem  to  effect  operator  performance  and  attention. 

Reliability  growth  programs  are  used  in  some  DoD  programs. 

If  the  program  manager  has  defined  a  reliability  growth  program,  the  projections  from  such 
growth  programs  should  not  be  used  as  part  of  the  operational  evaluation.  The  potential  growth 
of  system  reliability,  during  or  after  the  completion  of  the  operational  testing,  may  not  be  easy  to 
estimate;  it  is  dependent  on  resources,  dedication,  and  many  technical  details.  If  a  projection  is 
needed,  it  is  preferable  to  use  the  reliability  growth  experience  from  similar  systems  as  a  basis 
for  what  has  been  done  and  what  might  be  done  on  the  system  in  question.  A  projection  should 
never  be  reported  as  a  test  result  The  test  result  should  be  an  observed  value. 
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Maintainability  relates  to  the  ease  and  efficiency  of  performing  maintenance.  It  Is 
defined  as 

the  ability  of  an  Item  to  be  retained  In  or  restored  to  specified  condition 
when  maintenance  Is  performed  by  personnel  having  specified  skill  lev¬ 
els,  using  prescribed  procedures  and  resources,  at  each  prescribed 
level  of  maintenance  and  repair. 

There  are  three  important  dimensions  to  the  examination  of  system  maintainability.  The  first  is 
the  average  corrective  maintenance  time  required  to  restore  the  system  to  its  mission-capable 
condition.  This  maintainability  characteristic  gives  a  view  of  how  long  the  system  will  be  under 
repair  after  mission-critical  failures.  The  average  repair  time  for  a  system  might  be  two  hours. 
In  this  situation,  the  system  would  be  unavailable  due  to  maintenance  for  an  average  of  two 
hours  after  each  mission-critical  failure. 

The  second  dimension  addresses  the  maintenance  time  required  to  restore  the  system  after  any 
failure  that  requires  corrective  maintenance.  The  average  time  to  restore,  considering  all  correc¬ 
tive  maintenance,  may  be  longer  or  shorter  than  the  time  feu  mission-critical  failures. 

The  third  dimension  to  consider  is  the  manpower  required  to  perform  the  repair  function.  If  it 
takes  two  hours  for  the  average  repair,  there  is  a  considerable  difference  in  the  required  support 
resources  if  the  repair  requires  one  technician  or  three  technicians  for  this  two-hour  period. 

PARAMETERS 

A  maintainability  parameter  that  addresses  the  length  of  time  required  to  restore  the  system  to  a 
mission-capable  state  is  Mean  Operational  Mission  Failure  Repair  Tune  (MOMFRT). 

Total  number  of  clock  horns  of  corrective,  on-system. 

Mean  Operational  active  repair  time  used  to  restore  failed  systems  to 
MissionFailure  _  mission-capable  status  after  an  Operational  Mission  Failure 
Repair  Tune  Total  number  of  Operational  Mission  Failures 

A  parameter  that  addresses  the  time  required  to  restore  all  failures  is  Mean  Corrective 
Maintenance  Tune  (MCMT). 

Mean  Corrective  Total  number  of  clock  hours  of  corrective,  on-system. 
Maintenance  active  repair  time  due  to  all  corrective  maintenance 
Time  Total  number  of  incidents  requiring  corrective  maintenance 


MAINTAINABILITY 


Another  parameter  that  is  used  to  address  all  corrective  maintenance  time  is  Mean  Time  to  Re¬ 
pair  (Milk).  The  OTA  MOA  (see  Chapter  3)  has  a  number  of  definitions  for  MTTR.  One  defi¬ 
nition  is 


Mean  Time  To  Repair  = 


Sum  of  corrective  maintenance  times 
Total  number  of  corrective  maintenance  actions 
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In  addition  to  the  average  (or  mean)  time  that  is  required,  there  are  some  systems  where  it  is 
important  to  have  a  view  of  how  many  lengthy  maintenance  actions  there  will  be,  and  the  dura¬ 
tion  of  those  actions.  In  this  situation,  the  parameter  Maxinmm  Time  to  Repair  (MaxTTR)  may 
be  used.  Maximum  Time  to  Repair  will  give  a  time  that  a  specified  percent  of  the  maintenance 
actions  would  be  completed.  For  example,  a  MaxTTR  might  be  specified  for  90  percent  of  the 
failures.  If  the  MaxTTR  was  four  hours,  this  would  mean  that  90  percent  of  the  maintenance 
actions  would  be  completed  before  four  hours  elapsed. 

To  use  parameters  such  as  Mean  Corrective  Maintenance  Time  (MCMT),  it  is  important  to 
define  the  meaning  of  "corrective"  versus  "scheduled”  or  "preventive"  maintenance,  and  also  to 
define  when  each  of  the  important  time  measures  start  and  stop.  For  space  systems  and 
software-intensive  systems,  the  parameter  Mean  Time  To  Restore  Function  (MTTRF)  may  be 
used. 

Total  maintenance  time  to  restore  mission 
Mean  Time  To  _  functions  interrupted  by  critical  failures 
Restore  Function  —  Total  number  of  critical  failures 

This  parameter  addresses  both  scheduled  and  unscheduled  maintenance  time. 

Maintainability  parameters  that  address  the  manpower  resources  required  to  perform  mainte¬ 
nance  include  Maintenance  Manhours  per  Operating  Hour  (flight  hour,  mile,  round,  etc.).  For 
some  systems,  the  maintainability  (and  the  manpower  required)  is  measured  using  a  parameter  of 
Maintenance  Ratio  (MR),  which  is  a  measure  of  the  maintenance  manpower  required  to  maintain 
a  system  in  an  operational  environment.  It  is  expressed  as  the  cumulative  number  of  direct 
maintenance  man-hours  during  a  given  period  of  time,  divided  by  the  cumulative  number  of 
system  life  units  (such  as  hours,  rounds,  miles)  during  the  same  period.  There  could  be  a 
measure  of  MR  for  each  maintenance  level,  and/or  a  summary  for  a  number  or  all  levels.  The 
levels  of  maintenance  are  defined  and  labeled  differently  in  each  of  the  Services  (see  page  1-3). 
The  man-hours  considered  usually  include  all  types  of  maintenance,  e.g.,  corrective,  scheduled, 
preventive. 

Maintainability  also  may  include  an  examination  of  any  automated  system  diagnostics.  The 
label  "integrated  diagnostics"  addresses  all  forms  of  diagnostics  systems,  including  automatic, 
semi-automatic,  and  manual,  built-in  or  stand-alone,  in  an  integrated  fashion.  All  faults  and 
failures  must  be  diagnosed  by  some  method.  The  integrated  diagnostics  systems  should  be 
designed  and  tested  to  meet  this  need.  Integrated  diagnostics  is  discussed  in  section  22. 

The  maintainability  of  the  software  of  certain  systems  also  can  be  a  critical  issue.  This  topic  is 
discussed  in  section  2.5. 
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KEY  POINTS 

Maintainability  measurement  requires  a  reasonable  number  of  maintenance  events. 

Limiting  the  test  period  can  limit  the  applicability  of  the  maintainability  results.  To  obtain  a 
meaningful  evaluation  of  system  maintainability  requires  that  the  test  data  include  a  relatively 
good  cross-section  of  maintenance  actions  that  are  expected  to  occur  during  the  system's 
operational  service.  Examining  only  a  few  failures  may  exclude  insight  into  a  large  portion  of 
the  maintenance  training,  maintenance  documentation,  etc.  Sampling  a  number  of  maintenance 
actions  can  result  in  a  maintainability  evaluation  that  is  significantly  different  than  the  actual 
capability  of  the  system.  One  solution  is .  properly  structured  maintainability  demonstration,  but 
such  demonstrations  must  be  properly  planned  to  produce  results  that  are  valid  for 
supplementing  operational  test  results. 

Maintainability  demonstrations  can  be  used  in  OT&E  if  they  are  realistic. 

A  maintainability  demonstration  is  an  activity  wherein  maintenance  tasks  are  caused  to  be  per¬ 
formed  and  the  personnel,  test  equipment,  tools,  etc.,  involved  in  the  maintenance  are  evaluated 
by  observation.  Proper  operational  maintainability  demonstrations  must  be  structured  so  as  to 
present  a  representative  operational  environment  Maintainability  demonstrations  routinely  are 
used  as  part  of  the  contract  verification  requirements  imposed  on  the  developing  contractors,  but 
the  demonstration  methods  used  there  are  not  adequate  for  OT&E  because  they  lack  operational 
realism. 

If  maintainability  demonstrations  are  used  as  part  of  the  OT&E,  the  plan  for  these 
demonstrations  should  address  the  cross-section  of  maintenance  tasks  that  are  used,  the 
personnel  used  in  performing  the  maintenance,  and  die  conditions  surrounding  the  tasks  ured  in 
the  demonstrations.  Maintenance  events  or  tasks  that  are  performed  in  operating  units  usually 
include  events  that  are  not  the  result  of  failures.  Further,  there  arc  poorly  documented 
symptoms,  false  alarms  from  the  diagnostics  system,  and  other  types  of  tasks  that  may  not  be 
included  in  a  task  list  that  is  provided  by  the  contractor  or  the  government  program  manager. 

The  best  source  of  a  realistic  task  list  would  be  the  product  of  having  senior  maintenance 
technicians  from  the  relevant  operating  units  modify  the  contractor’s  task  list,  based  on  mainte¬ 
nance  tasks  required  for  similar  systems.  These  task  lists  can  be  used  to  identify  pre-faulted 
modules  that  have  been  used  as  part  of  maintainability  demonstrations.  The  personnel  used  in 
the  demonstrations  should  be  as  representative  of  the  expected  operational  personnel  as  possible. 

The  conditions  attending  the  demonstration  also  should  be  realistic.  It  should  not  be  conducted 
in  a  laboratory  if  the  actual  maintenance  will  be  done  in  the  field,  in  all  weather,  and  under  24- 
hour-a-day  lighting  conditions.  If  maintenance  tasks  are  to  be  performed  while  wearing  chemi¬ 
cal,  biological,  radiation  (CBR)  protective  clothing,  then  the  „re  of  this  clothing  should  be  con¬ 
sidered  when  planning  the  demonstration.  The  proximity  of  support  assets,  such  as  support 
equipment,  documentation,  etc.,  also  should  be  representative  of  what  might  exist  in  an  operating 
unit. 
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Built-in  test  equipment  and  other  diagnostics  systems  must  be  properly  tested  and  false 
alarm  rates  documented. 


Built-in  test  equipment  and  other  diagnostics  systems  must  be  tested  properly,  as  they  too  may  be 
fundamental  to  maintaining  the  system.  This  testing  must  not  be  glossed  over,  as  is  often  the 
case.  False  alarm  rates  frequently  are  not  discussed  in  test  documentation  and  therefore  make  it 
very  difficult  to  evaluate  the  BIT  responses  that  occur  during  OT  (see  section  2.2). 

Any  routine  scheduled  or  preventive  maintenance  should  be  carefully  examined  during 
OT. 

The  total  requirement  for  scheduled  or  preventive  maintenance  can  be  significant  for  some 
systems.  This  significance  may  exist  either  in  the  total  system  downtime  to  perform  the 
maintenance,  or  in  the  total  number  of  manhours  required.  Routine  maintenance  events,  such  as 
changing  oil  in  generator  power  supplies,  should  be  examined  for  adequate  accessibility.  The 
“reasonableness”  of  the  estimates  for  the  total  downtime  or  manhours  expended  in  these  areas 
should  be  examined.  The  time  required  for  each  of  the  significant  scheduled  maintenance  tasks 
should  be  evaluated  as  part  of  the  OT. 

The  time  for  off-equipment  repairs  can  be  significant 

One  factor  often  overlooked  in  requirements  and  test  documents  is  the  criteria  for  off-system  (or 
off-equipment)  repairs.  Mean  time  to  repair  (MTTR)  usually  applies  to  the  time  required  to 
return  the  system  to  a  mission-capable  state.  This  may  involve  a  change  in  a  major  system 
component  The  time  to  repair  applies  to  the  time  needed  to  isolate  the  failure  to  a  component 
and  to  change  the  failed  component  but  does  not  involve  the  actual  repair  time  to  return  the 
component  to  a  ready-for-issue  status.  MTTR  can  be  manipulated  by  a  "shotgun  maintenance” 
approach  (e.g.,  removing  and  replacing  the  three  or  four  most  likely  failure  candidates),  but  such 
actions  may  go  undetected  if  the  removed  parts  are  not  tracked  through  the  next  higher  level  of 
maintenance.  Further,  this  approach  causes  serious  logistics  concerns  due  to  the  quantities  of 
good  units  being  processed  in  die  repair  pipeline,  tying-up  maintenance  resources  later  labeled  as 
"Retest  Okay"  items.  It  also  may  have  a  major  affect  on  the  number  of  spares  and  how  they  are 
positioned. 

An  evaluation  that  addresses  only  on-equipment  MTTR  fails  to  consider  the  impact  on  the 
maintenance  organization  from  the  manpower  and  support  equipment  required  to  actually  repair 
the  item,  and  it  does  not  consider  the  impact  cm  the  logistics  system  of  the  material  needed  to 
make  the  eventual  item  repair.  Off-system  repairs  should  be  evaluated  to  determine  the  potential 
for  unexpected  levels  of  support  costs.  The  inability  to  logistically  support  the  system  once 
fielded  can  be  significant  To  avoid  this,  off-equipment  repair  should  be  addressed  in  the 
requirements  and  OT&E  documents. 

Unique  maintainability  characteristics  or  requirements  should  be  identified  and  included 
intheOT. 

The  system  being  tested  should  be  examined  to  determine  if  there  are  any  unique  maintainability 
characteristics  or  requirements.  Examples  might  be  maintenance  of  nuclear  hardness  features  or 
special  features  for  battle  damage  repair.  These  unusual  features  should  be  considered  when 
planning  the  maintainability  activities  for  the  OT. 
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Interoperability  is  defined  as 

the  ability  of  the  systems,  units,  or  forces  to  provide  services  to  and 
accept  services  from  other  systems,  units,  or  forces,  and  to  use  the 
services  so  exchanged  to  enable  them  to  operate  effectively  together. 

In  the  context  of  operational  suitability  and  OT&E,  interoperability  addresses  the  ability  of  the 
system  to  be  used  in  concert  with  other  types  of  systems  and  other  systems  of  the  same  type  that 
are  necessary  to  accomplish  the  required  mission  or  missions. 

Interoperability  is  frequently  considered  to  have  an  effect  on  both  the  suitability  and  the 
effectiveness  of  the  system.  Operational  testing  must  be  focused  on  those  supporting  or 
companion  systems  that  are  essential  for  the  system  under  test  to  meet  its  operational 
requirements.  During  the  early  test  planning,  the  critical  companion  systems  must  be  identified 
and  documented  by  the  Service  in  the  TEMP.  The  quality  of  the  interoperation  that  must  be 
examined  by  operational  testing  must  be  defined  in  derail  in  the  OT&E  test  plan  prepared  by  the 
OTA. 

Interoperability  recognizes  that,  for  the  system  to  perform  its  mission,  there  are  functions  that  are 
performed  by  two  or  more  items  in  concert  For  example,  data  and  commmfcations  systems 
must  have  the  technical  capability  to  interface  and  exchange  data.  Issues  for  these  systems  are 
usually  resolved  during  technical  testing,  but  operational  testing  may  result  in  additional 
confidence  in  the  interoperability  or  may  provide  additional  and  more  realistic  situations  under 
which  to  assess  the  operation  of  the  systems. 

PARAMETERS 

Interoperability  usually  is  evaluated  in  a  qualitative  manner.  However,  there  may  be  aspects  that 
are  described  quantitatively  and  therefore  contribute  to  the  presentation  of  the  system's 
interoperability  status. 

One  way  to  describe  the  interoperability  of  a  system  being  tested  is  to  discuss  what  limitations 
the  system  imposes  on  operations  when  it  is  used  with  other  systems.  This  would  entail 
preparing  the  following: 

•  A  list  of  those  systems  that  will  require  special  procedures  when  operated 
simultaneously  with  the  system  under  test,  and 

•  A  list  of  those  systems  whose  modes  of  operation  must  be  changed  when 
in  the  presence  of  the  tested  system. 


KEY  POINTS 

Supporting  or  companion  systems  need  to  be  identified  in  the  early  versions  of  the  TEMP. 

The  identification  of  supporting  or  companion  systems  is  one  of  the  keys  to  conducting  a 
successful  evaluation  of  the  system's  interoperability.  By  Milestone  t  the  systems  and 
equipments  that  win  have  critical  interoperation  with  the  system  under  test  should  be  identified 
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and  documented  by  the  Service  in  the  appropriate  acquisition  documents.  The  early  versions 
of  the  TEMP  should  highlight  these  companion  systems  and  the  need  to  have  the  required  test 
assets  to  verify  interoperability. 

The  consideration  of  companion  or  supporting  systems  also  should  address  other  systems 
that  are  under  development 

This  consideration  should  include  not  only  existing  systems,  but  also  companion  systems  that  are 
being  developed  at  the  same  time  as  the  system  being  examined.  There  are  frequently  problems 
in  acquiring  test  assets  when  systems  are  in  the  development  stage.  In  such  a  case,  conducting 
dual  tests  of  two  systems  that  are  both  being  developed  may  be  difficult,  but  the  OT&E  planning 
must  address  the  need  for  examining  any  critical  interoperability.  If  such  dual  testing  is  not 
possible,  then  the  evaluation  must  discuss  the  likelihood  of  potential  problems  and  the  limitations 
to  the  OT&E  because  the  two  systems  were  not  tested  simultaneously. 

Maturity  of  supporting  or  companion  systems  must  be  understood. 

The  relative  maturity  of  the  supporting  or  companions  systems  must  be  part  of  the  assessment  of 
what  is  needed  during  the  OT.  If  these  other  systems  are  not  mature  enough  to  provide  a  realistic 
level  of  :he  planned  support,  the  OT  may  provide  invalid  answers.  Also,  the  likely  maturity  at 
fielding  should  be  assessed.  If  the  suitability  of  the  system  when  fielded  is  dependent  on 
supportin'  systems,  e.g.,  targeting  information  systems,  and  Ac  supporting  systems  are  unlikely 
to  be  available  to  provide  the  requited  support,  then  this  potential  deficiency  needs  to  be 
highlighted  to  the  decisionmakers. 

Determination  of  adequate  suitability  depends  on  the  performance  of  the  supporting 
systems. 

When  the  suitability  of  the  system  under  test  is  being  determined,  the  acceptability  of  the 
supporting  systems  also  should  be  pan  of  the  judgment  An  example  might  be  a  targeting  system 
for  a  missile  system:  if  the  targeting  system  cannot  meet  its  projected  requirements,  then  the 
missile  system  can  hardly  be  expressed  as  being  suitable. 

Interoperability  problems  may  cause  system  limitations. 

One  category  of  interoperability  problems  is  a  situation  where  a  system  must  be  limited  in  its 
operation  due  to  the  proximity  to  another  system.  Examples  include  a  radio  transmitter  that  must 
be  turned  off  when  near  another  type  of  radio  or  communications  device;  an  aircraft  that  must  fly 
at  reduced  speeds  when  in  the  company  of  another  aircraft  type;  and  a  jammer  that  must  be 
turned  off  if  a  radar  is  to  work  or  if  a  certain  missile  is  to  be  fired.  Another  example  is  the  limi¬ 
tation  on  the  use  of  viewing  devices,  low-light-level  television,  etc.,  if  flares  are  to  be  used. 
Limitations  to  system  operation  that  are  die  result  of  interoperability  should  be  identified  during 
OT. 

Interoperability  should  be  addressed  in  the  OT&E  prior  to  Milestone  III. 

By  Milestone  II,  the  TEMP  should  indicate  the  manner  in  which  major  interoperability  areas  will 
be  examined  uiider  operational  testing.  The  test  resources  description  must  indicate  what 
companion  or  supporting  systems  ate  critical  to  the  system  in  question.  The  test  resource 
planners  need  to  assure  that  these  companion  or  supporting  systems  will  be  available  for  the  test 
period. 


Chapter  1 


Page  1-19 


Compatibility  is  defined  as 


the  capability  of  two  or  more  Items  or  components  of  equipment  or 
material  to  exist  or  function  In  the  same  system  or  environment  without 
mutual  Interference. 

Compatibility  addresses  and  includes  many  different  areas.  It  concerns  the  capability  of  the 
equipment  in  the  system  to  operate  with  each  of  the  required  supporting  equipments,  e.g., 
electrical  power  generation,  air  conditioning,  hydraulic  power  subsystem,  as  well  as  with  other 
elements  of  the  system.  It  also  addresses  the  interface  with  logistics  support  items,  including  test 
equipment,  servicing  equipment,  maintenance  stands,  handling  equipment,  and  elements  of  the 
transportation  systems  (see  section  1.7).  Compatibility  includes  physical,  functional,  electrical 
and  electronic,  and  environment  conditioning  areas.  Human  factors  (covered  in  section  1.13), 
environmental  factors  (section  2.3),  and  electromagnetic  environmental  effects  (E3)  (section  2.4) 
also  are  compatibility  considerations. 

Physical  compatibility  involves  attachment  pins,  connectors,  the  interconnecting  wires,  cables, 
alignment,  and  mechanical  linkages.  Physical  compatibility  may  involve  the  ability  to  install  the 
item  in  its  assigned  location,  physical  clearances,  and  item  volume.  These  physical 
characteristics  involve  compatibility  with  other  elements  of  the  operational  system,  as  well  as 
equipment  that  is  part  of  the  logistics  or  maintenance  environment  Electrical  or  electronic 
compatibility  considerations  include  voltage,  current  and  the  frequency  for  systems  using 
alternating  current  For  radio  frequency  or  visible  light  interfaces,  a  basic  consideration  is  the 
frequency  of  the  transmitted  signal.  Other  factors  considered  include  bandwidth,  frequency 
hopping  patterns,  and  signal  polarization.  Environmental  conditioning  considerations  address 
the  compatibility  of  heating  and  cooling  subsystems.  The  cooling  to  be  provided  for  electronics 
items  must  be  consistent  with  the  requirements  of  the  system. 

PARAMETERS 

Compatibility  parameters  involve  the  measurement  of  many  different  aspects  of  the  system's 
characteristics,  as  well  as  compatibility  by  function.  While  much  of  the  detailed  compatibility 
testing  is  the  domain  of  DT,  there  is  a  need  to  monitor  and  assess  compatibility  during  OT. 
Additional  or  expanded  environments  usually  are  present  during  the  conduct  of  a  realistic  OT, 
and  therefore  there  is  some  likelihood  that  problems  not  seen  during  DT  may  be  uncovered  in  the 
operational  testing. 

Some  of  the  parameters  that  may  be  considered  during  DT  and/or  OT  include: 

•  Physical  -  Attachment  pins  and  connectors,  alignment,  physical  dimensions,  volume, 
and  weight 

•  Electrical  -  Voltage,  cycles,  power  profile  or  stability,  and  surge  limits. 

•  Electronic  -  Frequencies,  modes,  rates,  control  logic,  and  telemetry. 

•  Environmental  Conditioning  -  Heating,  cooling,  shock  and  vibration  protection. 

•  Software  -  Formats,  protocols,  and  messages. 
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•  Hardware/Software  -  Conventions,  standards,  timing,  sequencing,  sensing,  and 
control  logic. 

•  Data  -  Rates,  inputs,  characters,  and  codes. 

KEY  POINTS 

DT  results  may  help  focus  OT  planning. 

Compatibility  requirements  should  be  monitored  during  the  early  development  stages  of  the 
acquisition  process  to  ensure  all  compatibility  areas  and  issues  are  addressed.  DT  results  should 
be  tracked  to  assist  in  the  OT  planning  and  to  avoid  duplication  of  testing  efforts. 

Early  operational  testing  may  indicate  unforeseen  compatibility  problems. 

The  early  OT  phase  should  include  a  compatibility  issue  to  ensure  that  operational  considerations 
introduced  into  the  testing  at  this  point  are  not  causing  problems  that  were  not  observed  in  DT. 
Identification  of  potential  problems  not  anticipated  by  die  designer  could  include  electrical 
power  variations,  unexpected  electrical  interference,  or  the  need  for  additional  air  conditioning. 

Nominal  operations  may  not  expose  incompatibilities. 

Nominal  operations  may  not  expose  interference  or  incompatibility  problems,  and  special  tests 
may  be  required  to  test  the  system  in  various  modes  and  operational  extremes  to  detect  potential 
interference. 

Operational  test  personnel  most  address  the  needs  for  any  special  resources/systems  that 
are  required  for  compatibility  testing. 

If  special  facilities,  instrumentation,  and  simulators  for  compatibility  are  required  for  OT, 
advanced  planning  is  required.  These  requirements  must  be  discussed  in  early  versions  of  the 
TEMP.  Failure  to  do  advanced  planning  may  result  in  a  delay  to  die  operational  testing,  or  in 
conducting  the  test  without  completing  some  of  die  objectives. 

Modifications  or  upgrades  may  introduce  compatibility  problems. 

The  addition  of  new  or  advanced  capabilities  to  a  weapon  system  may  introduce  the  potential  fin- 
compatibility  problems.  For  example,  if  a  system  is  upgraded  with  more  advanced  computer  and 
electronics  systems,  the  original  environmental  equipment  may  be  unable  to  provide  adequate 
cooling  to  tire  new  system.  As  a  result,  the  new  electronics  will  operate  in  higher  temperatures 
and  be  less  reliable  than  projected  when  used  by  the  operating  units.  A  similar  situation  may 
result  with  a  ground  combat  vehicle  that  is  upgraded  with  heavier  weapons  and,  as  a  result,  has  a 
weight  that  may  be  incompatible  with  elements  of  the  drive  train,  brakes,  etc.  The  integration  of 
non-developmental  items  (NDI)  into  a  system  also  may  introduce  compatibility  problems. 

Compatibility  of  procedures  can  be  a  factor  in  system  performance. 

The  compatibility  of  two  systems  may  depend  to  a  large  degree  on  the  procedures  being  used  and 
bow  the  procedures  are  followed.  During  tire  testing  of  one  system,  it  was  discovered  that  the 
main  system  was  not  compatible  with  the  command  and  control  system.  The  system  was  fired 
and  controlled  in  a  fully  automated  manner,  while  the  fire  support  system  was  manually 
operated.  The  crews  that  were  to  make  the  manual-to-automated  translations  could  not 
communicate  inside  some  of  tire  shelters  because  of  the  noise  level  that  resulted  from  tire  support 
equipment  (400-cycle  cooling  fans,  and  turbine  generators). 
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Supportability  Is  defined  as 


the  degree  to  which  system  design  characteristics  and  planned  logistics 
resources,  Including  manpower,  meet  system  peacetime  readiness  and 
wartime  utilization  requirements. 

This  element  addresses  the  balance  between  the  system's  support  needs,  which  are  a  result  of  the 
system  design,  and  the  planned  logistics  support  for  the  system. 

Other  suitability  elements  also  address  aspects  of  this  balance,  e.g.,  reliability,  maintainability, 
manpower,  documentation,  training,  and  the  like.  The  scope  of  logistics  supportability  within 
the  OT&E  documentation  is  limited  to  those  aspects  that  are  not  covered  under  other  topics.  It 
also  includes  the  integrated  aspects  of  the  logistics  planning.  Key  items  that  should  be  addressed 
under  this  element  include  supply  support  (the  planned  numbers  and  placement  of  spares  and 
repair  parts),  test  or  support  equipment  fen  all  levels  of  maintenance,  and  planned  support 
facilities. 

Some  systems  go  into  operational  testing  without  a  plan  for  specific  numbers  of  spares  or  repair 
parts.  This  situation  becomes  a  test  limitation.  The  problem  is  due  in  large  part  to  cost  and 
manufacturing  constraints  imposed  either  on  or  by  the  developer.  In  those  cases,  the  contractor 
provides  a  portion  of  the  required  system  support  package. 

PARAMETERS 

Logistics  supportability  is  most  often  evaluated  in  a  qualitative  manner,  although  there  may  be 
quantitative  factors  that  are  used  in  some  of  the  elements  of  supportability.  For  example,  supply 
support  might  be  assessed  by  examining  parameters  such  as  percent  of  items  in  local  supply 
assets,  fill  rates,  etc.  These  parameters  are  then  judged  qualitatively  to  arrive  at  the  assessment 
of  the  entire  subject  of  logistics  supportability.  The  evaluation  of  supportability  is  intended  to 
consider  all  these  subordinate  factors,  or  considerations,  and  provide  a  composite  evaluation  of 
the  balance  between  the  support  that  is  needed  and  the  support  that  is  planned. 

In  some  test  programs,  the  area  of  logistics  supportability  also  is  used  to  cover  other  suitability 
areas  (e.g.,  transportation,  manpower  supportability,  documentation,  or  training)  that  are  not 
significant  enough  for  a  particular  system  to  warrant  an  individual  test  objective. 

KEY  POINTS 

Early  ILS  planning  can  be  assessed  as  part  of  logistics  supportability  evaluation. 

Early  activities  in  Integrated  Logistics  Support  (ILS)  planning  may  begin  before  Milestone  L 
Some  aspects  of  Logistics  Support  Analysis  (LSA)  also  are  conducted  in  this  time  frame. 
Review  of  the  products  of  these  activities  may  be  key  to  any  early  operational  assessment  of 
operational  suitability.  These  early  activities  also  will  give  some  information  on  the  criticality  of 
the  support  aspects  of  the  system.  If  the  system  is  to  have  a  unique  support  concept  that  will 
succeed  only  under  certain  system  reliability  and  maintainability  levels,  then  this  might  be 
identified  as  an  operational  suitability  critical  operational  issue  (COI).  Such  an  examination  will 
begin  to  focus  on  the  critical  aspects  of  the  suitability  that  need  to  be  addressed  in  the  OT&E 
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planning.  One  example  of  potential  support  problems  is  a  system  that  is  intended  to  support  a 
number  of  different  weapon  systems  -  within  the  context  of  one  weapon  system,  die  planning 
may  be  accurate,  but  for  others  or  for  the  total  support  requirement,  the  planning  may  not  be 
correct. 

n.S  planning  can  provide  the  basis  to  assess  the  planned  logistics  support. 

An  early  assessment  of  logistics  supportability  can  be  made  by  reviewing  the  status  of  die 
logistics  support  planning.  One  portion  of  this  activity  should  be  a  review  of  the  Integrated 
Logistics  Support  Plan  (ILSP),  or  of  the  results  of  the  logistics  support  analysis.  How  complete 
and  well  done  the  planning  is  done  determines  to  a  large  degree  how  acceptable  die  support  of 
the  system  will  be.  Modeling  and  simulation  (see  section  2.1)  also  may  be  used  to  analyze  the 
ability  of  the  system  to  meet  some  of  its  suitability  requirements  with  the  planned  level  of 
support. 

Test  planning  most  address  the  support  for  the  items  under  test 

As  die  operational  test  is  planned,  the  support  assets  (e.g.,  spares,  test  equipment,  support 
equipment)  must  be  identified  in  the  TEMP  and  in  die  detailed  test  plan  and  must  be  in  place  for 
the  conduct  of  die  operational  test  If  these  support  assets  are  not  available,  it  will  be  more 
difficult  to  assess  the  operational  suitability  of  the  system,  and  it  will  be  impobible  to  evaluate 
the  performance  of  the  support  items. 

Operational  test  data  should  be  compared  to  the  ILS  planning  factors. 

Assessment  of  the  ILS  planning  should  continue  as  pan  of  the  OTA’s  activity  to  support  the 
OT&E  repiort  for  the  Milestone  III  decision.  Operational  test  data  should  be  compared  to  the  ILS 
planning  factors  and  evaluated  to  determine  if  the  planning  factors  reflect  the  real  needs  of  die 
system.  If  the  test  data  contain  demand  rales  (e.g.,  MTBR)  horn  the  operational  test  period, 
these  rates  should  be  compared  to  the  rates  that  were  projected  in  the  logistics  planning. 
Incompatibility  of  the  planning  and  the  system  parameters  is  one  of  the  major  causes  of  systems 
being  unable  to  meet  their  availability  requirements  during  the  first  stages  of  operational  fielding. 

Supportability  of  software  should  be  considered. 

For  those  systems  that  have  significant  software  elements,  the  support  of  the  system's  software 
can  be  an  important  factor  in  die  ability  to  provide  logistics  support  for  the  entire  system. 
Planning  for  support  of  software  includes  both  manpower  and  equipment  resources.  The 
personnel  who  will  be  used  to  maintain  or  upgrade  the  software  during  its  operational  use  must 
have  adequate  documentation  to  allow  them  to  pierform  their  function.  Evaluations  during 
OT&E  can  provide  insight  into  the  adequacy  of  the  planning  for  the  support  of  the  software. 
(Software  supportability  is  discussed  in  more  detail  in  section  2.5.) 

Supply  support  during  operational  testing  may  be  unrealistic 

Some  test  programs  provide  an  "iron  mountain"  of  spares  and  repair  parts  to  optimize  the  use  of 
valuable  test  range  times,  lest  forces,  instrumentation,  etc.  In  these  situations,  the  evaluation  of 
test  results  must  compensate  for  die  unrealistic  conditions  in  the  logistics  support  of  the  systems 
under  test 
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Transportability  is  defined  as 

the  capability  of  materiel  to  be  moved  by  towing,  self-propulsion,  or 
carrier  through  any  means,  such  as  railways,  highways,  waterways, 
pipelines,  oceans,  and  airways.  (Full  consideration  of  available  and 
projected  transportation  assets,  mobility  plans  and  schedules,  and  the 
Impact  of  system  equipment  and  support  Items  on  the  strategic  mobility 
of  operating  military  forces  Is  required  to  achieve  this  capability.) 

System  requirements  and  employment  methods  may  dictate  that  specific  transportation  modes  be 
used  for  deployment  purposes  for  some  deployment  scenarios.  Assessment  of  these  different 
modes  of  transportation  should  be  accomplished  by  the  developing  agency.  Other  areas  of 
attention  include  the  need  for  any  unusual  transportation  or  handling  equipment.  The 
compatibility  with  transport  aircraft,  ships,  or  any  vehicles  that  ate  essential  for  the  system  to 
arrive  at  its  destination  and  then  perform  its  mission  also  most  be  addressed.  This  compatibility 
includes  physical  dimensions  and  clearances,  tie  downs,  and  load  capacity.  Routine 
transportation  and  mobility  movement  of  spates  and  support  equipment  also  should  be  consid¬ 
ered.  The  capability  of  the  item  to  be  included  in  any  required  at-sea  replenishment  or  amphibi¬ 
ous  operations  should  be  addressed.  These  elements  include  the  ability  of  the  system  to  be 
transported  by  the  planned  means  or  within  the  intended  transportation  capability  of  the  DoD. 

Transportability  also  may  include  the  "deployability"  of  a  system  or  equipment  that  is  deployed 
with  combat  units  to  the  combat  area,  and  the  “portability”  of  items  that  are  carried  by  user  per¬ 
sonnel  during  use.  Compatibility  with  the  using  personnel  also  must  be  assessed.  Personnel 
who  will  prepare  and  move  die  item  as  part  of  its  transportation  must  be  physically  capable  of 
performing  the  required  tasks. 

PARAMETERS 

Parameters  associated  with  transportability  must  address  the  characteristics  that  will  allow 
existing  transportation  assets  to  move  and  transport  the  equipment  as  needed  to  support  the 
operational  mission.  If  new  transportation  assets  are  planned  specifically  for  this  system,  then  the 
evaluation  should  address  the  acceptability  of  these  new  assets.  Some  parameters  associated 
with  transportability  are: 

•  Does  the  using  organization  have  die  provisions  for  handling  and 
transporting  the  system? 

•  Can  the  system  be  transported  to  the  theater  by  the  preferred  means? 

•  Can  the  system  be  moved  adequately  within  the  theater  of  operations? 

•  Are  the  dimensions  and  weight  within  the  required  limits  for  each  possible 
transportation  mode  that  will  be  required  to  move  die  equipment? 
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KEY  POINTS 

Unique  transportability  requirements  should  be  identified. 

Initial  system  transportability  requirements  should  be  specified  in  the  early  system  planning 
documents,  and  the  acquisition  and  using  agencies  should  assess  these  requirements  against  the 
capabilities  of  existing  transportation  assets.  The  role  of  OT  planners  is  to  examine  these 
requirements  and  determine  if  test  resource  assets  (e.g.,  cargo  aircraft,  rail  cars)  are  needed  for 
OT.  The  need  for  special  transportability  test  events  also  should  be  addressed.  If  there  is  a 
critical  transportability  issue,  this  needs  to  be  identified  in  an  early  TEMP. 

Transportability  of  the  system  should  be  verified  as  part  of  operational  testing. 

If  the  system  is  a  major  rail-  or  air-transportable  item,  for  example,  a  tank  or  a  helicopter,  the 
compatibility  with  die  transporting  means  should  be  verified.  Often,  the  DT  will  examine  this 
compatibility,  but  from  a  technical  standpoint,  ie.,  does  it  “fit?”  While  an  argument  can  be 
made  that  the  transportability  requirement  has  been  verified  by  this  DT  evaluation,  there  are 
operational  factors  that  need  consideration.  Just  because  contractor  engineers  are  able  to  load 
the  system  into  the  transporting  aircraft  on  a  clear  day  in  good  weather  does  not  mean  that  the 
using  troops  can  perform  the  same  task  under  all  weather  and  lighting  conditions.  In  the  case  of 
manpack  items,  the  ability  of  the  person  to  carry  the  item  may  be  proven,  but  can  the  person  who 
is  carrying  the  item  still  perform  the  assigned  mission,  or  is  there  some  negative  impact  on  com¬ 
bat  effectiveness.  The  OT  examination  is  directed  more  at  "can  it  be  dene  by  the  normal  user 
troops  under  the  conditions  predicted  for  die  using  organization."  Operationally  realistic 
conditions  can  yield  results  different  from  those  produced  by  the  DT. 

All  projected  areas  of  operations  should  be  included  in  the  transportability  assessment. 

Due  to  weight,  dimensions,  and  system  characteristics,  die  transportability  of  die  system  may  be 
limited  in  certain  geographic  areas  of  operation.  For  example,  the  dimensions  of  a  tank  may  be 
compatible  with  U.S.  rail  transportation,  but  the  use  of  rail  transportation  in  other  countries  will 
be  limited  because  rail  widths  and  capability  differ  from  that  in  the  United  States.  Systems  with 
extensive  global  commitments  must  be  analyzed  very  carefully  to  ensure  that  all  transportability 
requirements  are  understood  and  can  be  met.  If  the  system  has  a  unique  transportability 
requirement,  it  should  be  made  part  of  the  system  planning  documentation  and  considered  for 
examination  as  part  of  the  OT. 

Transportability  should  indude  the  movement  of  the  system  into  combat  locations. 

The  system  must  be  moved  into  and  within  a  theater  of  operations  consistent  with  its  mission. 
This  issue  may  deal  with  airplane,  train,  or  ship  loading  and  internal  or  external  helicopter  loads. 
The  examination  should  address  die  ability  of  the  transporting  system  to  carry  the  load,  and  any 
impacts  on  maneuverability  once  loaded.  It  should  ensure  that  the  weight  and  dimensions  of  the 
new  system  can  be  supported  by  the  transportation  network  and  current  bridging  (to  include 
tactical  bridging)  in  the  required  operational  environment 

Testing  of  systems  after  being  transported  can  be  critical  for  some  systems. 

For  some  systems,  operational  testing  should  be  planned  to  verify  the  fact  that  transporting  the 
system  has  not  degraded  its  capability.  Realistic  scenarios  of  preparation  for  transport  the  actual 
transport  of  the  system,  and  set-up  for  operation  all  should  be  included  in  the  operational  testing 
scenario.  The  importance  of  this  activity  varies  from  system  to  system,  and  should  be  included 
in  the  testing  requirements  if  the  requirement  is  judged  to  be  significant 
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Documentation  is  a  portion  of  technical  data  and  information  that  is  part  of  every 
system. 

For  the  purposes  of  OT&E,  documentation  comprises  operator  and 
maintenance  Instructions,  repair  parts  lists,  and  support  manuals,  as 
well  as  manuals  related  to  computer  programs  and  system  software. 

The  ability  to  operate  and  maintain  new  and  advanced  systems  can  be  highly  dependent  on  the 
completeness  and  accuracy  of  the  documentation  that  is  provided  with  the  system.  For  complex 
systems,  the  operator  and  maintainer  documentation  can  make  the  difference  between  success 
and  failure  of  the  system. 

During  the  documentation  development  process,  usually  during  the  full-scale  development 
phase,  representatives  of  the  user  organizations  generally  will  conduct  a  validation  of  the 
documentation.  The  validation  process  addresses  the  ability  to  locate  procedures  and  tasks,  as 
well  as  the  need  for  any  additional  tasks  required  to  support  maintenance  operations. 
Documentation  clarity,  accuracy,  and  ability  to  support  projected  skill  levels  also  is  validated. 
Cautions,  warnings,  and  advisories  are  reviewed  to  ensure  that  they  are  appropriate  for 
incorporation  in  the  manual,  and  are  checked  to  ensure  that  they  are  accurate,  dear,  and  easily 
identifiable  to  the  reader.  Preventive  maintenance  checks,  services,  and  procedures  also  ate  vali- 

The  validation  process  generally  is  accomplished  in  two  phases.  Developmental  testing 
personnel  usually  perform  the  first  phase,  and  determine  if  the  drawings,  figures,  specifications, 
and  procedures  are  technically  correct  The  second  phase  usually  is  done  by  operational  testing 
personnel.  They  determine  if  die  maintenance  technician  and  operator  can  understand  and 
correctly  perform  die  procedures  outlined  in  the  documentation.  This  examination  may  be 
performed  in  conjunction  with  other  suitability  evaluation  activities,  including  data  collection  for 
maintainability,  training,  and  human  factors  evaluations. 

In  addition  to  any  "formal"  validation  tasks,  the  documentation  also  should  be  part  of  the  activity 
on  other  OT  tasks.  For  example,  when  operations  or  maintenance  is  performed  during  die  OT, 
the  documentation  that  is  used  should  be  assessed  for  its  completeness,  accuracy,  case  of  use, 
etc.  The  OT&E  plan  should  discuss  how  the  results  of  these  naturally  occurring  documentation 
assessments  will  be  recorded. 

PARAMETERS 

Documentation  evaluation  is  primarily  qualitative  in  nature.  There  are  some  quantitative 
parameters  that  might  contribute  to  organizing  and  managing  the  assessment  of  the 
documentation.  Three  examples  follow. 

Percent  of  Critical  Tasks  or  Procedures  Available:  Clearly,  die  weakest  documentation 
procedure  is  the  one  that  does  not  exist  This  parameter  indicates  how  complete  the 
documentation  is  at  the  time  of  OT.  Documentation  can  be  made  available  in  various  forms, 
e.g.,  (baft,  "blue  line,"  final  deliverable.  The  percentage  of  each  form  that  is  provided  to  the 
operational  test  activity  can  be  a  good  indication  of  die  status  of  the  documentation  development 


Page  1-26 


Chapter  1 


Percent  of  Critical  Tasks  or  Procedures  Validated:  For  the  total  number  of  tasks  or  procedures  in 
the  operating  or  maintenance  manuals,  this  represents  the  percentage  that  have  been  validated.  It 
can  be  applied  to  maintenance,  operator,  or  other  support-critical  tasks  or  procedures,  and  it 
should  approach  100  percent  as  the  system  nears  the  production  decision. 

Percent  of  Erroneous  Procedures  or  Tasks:  For  the  total  number  of  tasks  and  procedures 
demonstrated,  this  is  the  percentage  that  are  considered  to  be  erroneous.  Other  similar 
parameters  can  be  developed  to  highlight  the  number  of  unclear  tasks,  tasks  that  have  too  much 
detail,  insufficient  detail,  etc.  The  parameters  will  depend  on  the  manner  of  collecting  this 
information  at  the  operational  test  site. 

KEY  POINTS 

Documentation  should  be  available  for  the  operational  test  phase. 

The  documentation  development  and  assessment  schedule  must  be  compatible  with  scheduled 
operational  test  time  frames.  The  early  assessment  of  the  documentation  preparation  program  is 
one  form  of  early  suitability  assessment  A  good  process,  with  adequate  schedule  and  resources, 
may  produce  good  documentation.  A  poorly  scheduled  program,  or  one  with  inadequate 
resources,  probably  will  yield  poor  results.  At  a  minimum,  preliminary  documents  should  be 
available  fen  the  operational  testing  phase,  even  if  the  final  documents  are  not  ready. 

Documentation  may  not  be  available  for  the  operational  testing  schedule. 

While  the  best  test  of  documents  is  to  use  them  during  OT,  the  delivery  dates  for  die 
documentation  may  make  it  difficult  to  evaluate  die  final  product  during  OT.  The  effect  of  any 
schedule  shortcomings  should  be  known  well  in  advance  and  arrangements  made  for  work¬ 
arounds,  use  of  draft  documents,  or  other  alternative  evaluations.  For  accelerated  programs,  the 
OTA  should  identify,  and  include  in  die  OT&E  plan,  alternative  methods  for  achieving  the 
evaluation  of  this  area.  Delays  in  availability  of  essential  technical  manuals  can  cause  a  test  to 
experience  disruptive  delays  or,  more  significandy,  result  in  an  improper  evaluation  of  the 
planned  support  system  or  system  reliability,  availability,  and  maintainability. 

Assessment  of  documentation  may  be  in  a  separate  test  phase. 

The  assessment  of  the  documentation  usually  requires  a  separate  documentation  test  phase  to 
determine  their  adequacy,  prior  to  using  the  documentation  as  part  of  system  operational  testing. 
This  testing  should  stress  the  use  of  military  personnel  skills,  tools,  facilities,  and  support 
equipment  that  are  planned  for  use  in  the  support  environment  when  the  system  is  fielded. 

Only  a  sample  of  the  operation,  maintenance,  and  support  tasks  in  the  documentation  may 
be  naturally  occurring  in  OT. 

One  of  die  difficulties  in  documentation  assessment  is  that  it  needs  to  address  a  relatively  large 
percentage  of  the  operating  and  maintenance  tasks  at  the  organizational  and  intermediate  (direct 
support)  levels,  as  well  as  a  sampling  of  maintenance  tasks  at  die  general  support  level.  It  may 
be  difficult  to  evaluate  voluminous  documentation  when  the  test  period  is  relatively  short  The 
use  of  system  documentation  during  operational  testing  will  provide  data  on  its  acceptability  in  a 
narrow  range  of  circumstances.  One  alternative  is  to  use  data  from  ocher  sources,  e.g., 
maintenance  during  DT  and  maintainability  demonstrations  (see  section  13).  Any  critical  task 
or  procedure  not  observed  adequately  dunng  the  operational  testing  should  be  validated  in  a 
separate  scheduled  event 
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Manpower  supportablllty  is  defined  as 

the  identification  and  acquisition  of  military  and  civilian  personnel  with 
the  skills  and  grades  required  to  operate  and  support  a  materiel  system 
over  Its  lifetime  at  peacetime  and  wartime  rates. 

Within  the  context  of  operational  test  and  evaluation,  manpower  supportability  takes  into 
consideration  the  numbers,  skill  types,  and  skill  levels  of  the  personnel  required  to  operate, 
maintain,  and  support  the  systems  in  both  peacetime  and  wartime  environments.  Manpower 
supportability,  therefore,  is  closely  related  to  training  requirements,  human  factors,  and 

maintainability. 

Increases,  deletions,  and  changes  to  the  force  structure  may  be  required  to  place  the  system  in  its 
operating  units.  Documents  are  developed  which  indicate  the  projected  manpower  requirements 
and  skill  codes  and  grades  that  are  necessary.  The  OT  objective  here  is  to  assess  if  the  projected 
levels  are  adequate  to  operate  and  support  the  system. 

The  determination  of  the  number  of  manpower  spaces  required  is  based  on  the  various  scenarios 
in  which  the  system  will  be  used.  They  depend  on  the  system  having  the  projected  reliability  and 
maintainability,  and  die  support  resources,  diagnostics,  test  equipment,  etc.,  as  planned. 
Shortfalls  in  these  areas  can  have  significant  impact  on  the  required  manpower.  If  the  projected 
manpower  is  inadequate  to  support  the  system,  then  significant  problems  could  occur  when  the 
system  is  fielded. 

PARAMETERS 

The  general  parameter  for  manpower  supportability  is  the  number  of  personnel  required  to  man 
the  system  when  it  is  employed.  It  addresses  operating,  maintenance,  and  other  support 
personnel,  and  their  required  skills  and  training.  Other  parameters  dial  might  be  used  include  the 
following  three: 

Cew  Size:  The  number  of  people  required  to  operate  die  system  and  perform  the  tasks  required 
in  each  speciality  and  at  each  skill  level,  and  to  use  the  system  in  the  intended  scenariofs). 

Maintenance  Ratio:  The  ratio  of  maintenance  manhours  per  operating  hour  or  life  unit  (see 
section  1.3).  This  measure  allows  the  comparison  of  the  projected  maintenance  workload  and 
the  workload  demonstrated  during  operational  testing.  On-  and  off-system  Maintenance  Ratios 
(MRs)  are  estimated  for  each  level  of  maintenance  and  for  each  skill  code.  MR  criteria  typically 
are  found  in  the  requirements  document 

Current  Crew  Size  and  Maintenance  Ratio:  A  comparison  of  the  system’s  manpower  require¬ 
ments  with  those  of  the  current  system  can  provide  a  meaningful  measure  of  the  criticality  of  the 
manpower  supportability. 
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KEY  POINTS 

Manpower  supportability  includes  examination  of  the  operating  crew. 

While  manpower  supportability  may  highlight  the  support  aspects  of  manpower  (e.g.,  mainte¬ 
nance  crews  and  skills),  the  operating  crew  is  an  important  part  of  the  evaluation.  The  OT&E 
test  plan  always  should  indicate  that  the  operating  crew  size,  skills,  etc.,  will  be  part  of  the 
evaluation. 

Manpower  deficiencies  actually  may  reside  in  other  suitability  areas. 

On  some  systems,  it  may  appear  that  the  manpower  estimated  is  inadequate  to  operate  or 
maintain  the  system.  If  the  system  is  experiencing  shortfalls  in  expected  reliability, 
maintainability,  or  diagnostics,  or  has  human  factors  problems,  then  the  deficiency  may  be  more 
correctly  assigned  to  these  areas. 

Manpower  planning  for  OT&E  should  not  indude  "Golden  Crews." 

The  personnel  who  man  and  maintain  the  systems  during  the  operational  testing  should  not  be  of 
such  a  high  skill  level  that  the  test  results  are  invalid.  The  thrust  of  OT&E  is  to  see  if  the  system 
is  suitable  in  the  hands  of  "troops"  who  are  representative  of  the  intended  operational  users. 
Realism  also  may  be  lost  if  the  personnel  in  the  testing  organization  have  received  a  greater  num¬ 
ber  of  exposures  to  the  tasks  than  will  be  the  case  with  the  planned  user  personnel.  An  example 
would  be  the  use  of  hand-held  ground-to-air  missiles  -  if  the  intended  users  will  never  have  the 
opportunity  to  fire  a  live  round  during  their  training,  but  only  have  exposure  to  simulators,  then  it 
is  unrealistic  to  use  personnel  in  the  OT  who  have  had  experience  with  a  number  of  firings 
during  DT,  or  other  testing  of  this  system,  or  similar  systems. 

Skill  levels  and  numbers  may  be  hard  to  evaluate. 

The  complement  of  personnel  that  is  used  during  some  operational  tests  may  have  somewhat 
higher  skill  levels  than  are  planned  for  fee  operational  units.  These  higher  skill  levels  are 
justified  by  the  OTA  as  necessary  because  it  takes  higher  skill  level  personnel  to  recognize 
deficiencies  in  the  system.  In  this  situation,  the  evaluation  of  the  manpower  resources  needed  to 
operate  and  maintain  the  system  must  consider  the  skills  that  were  present  during  testing.  The 
OT&E  plan  must  include  an  adequate  evaluation  procedure. 

Proper  manning  levels  for  systems  are  critical  for  efficient  operations. 

The  OTA  must  carefully  evaluate  the  manning  levels  of  units  who  will  field  new  systems. 
Improperly  manned  equipment  will  result  in  poorly  operated  and  maintained  systems.  During  fee 
fielding  of  one  system,  it  was  discovered  that  the  system  was  not  properly  manned;  additional 
personnel  were  required  to  allow  units  in  the  field  to  be  more  self  sufficient  On  another  major 
system,  the  proposal  maintenance  organizations  did  not  have  personnel  or  a  supporting  organi¬ 
zation  to  maintain  the  required  radio  frequency  signal  management  system.  This  lack  of  man¬ 
power  would  have  resulted  in  those  maintenance  tasks  being  transferred  to  a  higher  maintenance 
level,  thereby  reducing  the  efficiency  of  the  unit 
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1,10  TR&mrn® 


Training  and  Training  Support  are  defined  as 

the  processes,  procedures,  techniques,  training  devices,  and  equipment 
used  to  train  civilian  and  active  duty  and  reserve  military  personnel  to 
operate  and  support  a  materiel  system.  This  includes  Individual  and 
crew  training;  new  equipment  training;  Initial,  formal,  and  on-the-job 
training;  and  logistic  support  planning  for  training  equipment  and 
training  device  acquisitions  and  Installations. 

During  the  OT&E,  the  planned  training  program,  along  with  any  training  devices  and  equipment, 
should  be  evaluated.  The  training  program  and  supporting  materials  are  developed  during 
various  phases  of  the  weapon  system  development  process.  The  supporting  materials  include 
programs  of  instruction,  on-the-job  training  documentation,  training  materials,  and,  when 
required,  training  aids  and  simulators.  Training  materials  are  provided  for  both  individual 
operator  and  maintainer  training,  as  well  as  for  "collective"  training  for  crews  or  units. 

Operators  are  trained  to  perform  all  critical  tasks  required  to  operate  the  system.  Maintenance 
personnel  are  trained  to  perform  all  critical  tasks  required  to  maintain  the  system.  Collective 
training  is  provided  to  system  crews  whose  members  are  required  to  perform  as  a  team.  All  tasks 
must  be  accomplished  to  prescribed  training  standards.  If  tasks  to  be  performed  are  linked  to  or 
dependent  on  other  tasks  (e.g.,  firing  sequence,  or  an  initialization  sequence),  all  tasks  must  be 
performed  to  standard  in  a  single  performance  test. 

Evaluation  of  training  should  address  the  effectiveness  of  the  training  program  in  providing 
personnel  who  can  operate  and  maintain  the  system.  Maintenance  training  must  be  analyzed 
from  the  organizational  through  the  intermediate  level  and  include  the  training  program,  as  well 
as  training  aids,  simulators,  and  support  equipment  used  at  each  of  the  levels  of  maintenance. 
Particular  attention  always  should  be  given  to  performance  of  critical  tasks.  In  addition,  tasks 
that  are  new,  unique,  or  hazardous  must  be  included  in  the  evaluation,  or  some  assurance  should 
be  given  that  these  tasks  can  be  performed  satisfactorily. 

A  training  deficiency  exists  when  the  training  provided  does  not  address  the  skills  needed  to 
operate  or  maintain  the  system.  Once  a  deficiency  has  been  identified,  an  assessment  should  be 
made  that  classifies  the  deficiency  as  a  shortcoming  in  the  training  provided,  in  the 
documentation,  or  in  the  system  itself. 

PARAMETERS 

Training  effectiveness  is  based  on  both  the  training  programs  and  the  performance  of  the 
individuals  while  accomplishing  tasks  associated  with  the  use,  operation,  and  support  of  the 
system,  to  include  individual  and  collective  training. 

The  operational  evaluation  addresses  how  well  the  trained  individuals  perform  the  required  tasks. 
The  ability  to  perform  the  necessary  tasks  correctly,  once  the  individual(s)  is  in  an  operating 
environment,  establishes  that  the  system  can  be  operated  and  maintained  by  the  personnel  so 
trained.  However,  criteria  also  are  needed  for  operator  and  maintainer  performance  during 
combat.  These  criteria  may  not  be  the  same  as  those  used  during  peacetime.  For  example, 
changing  a  tank  engine  in  a  maintenance  bay  is  much  different  from  changing  the  engine  in  the 
field,  at  night,  and  under  rainy  conditions. 
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The  parameter  “critical  tasks  demonstrated”  is  the  ratio  of  critical  tasks  demonstrated  by  the 
trainee  using  validated  procedures  within  the  time  standard,  to  the  total  number  of  tasks 
attempted,  or  total  tasks  within  the  manuals.  It  can  be  calculated  for  each  maintenance  level  or 
for  each  skill  category  (MOS,  AFSC,  etc.).  This  parameter  has  a  close  alignment  with  some  of 
the  parameters  used  in  the  evaluation  of  documentation  (see  section  1.8). 

KEY  POINTS 

OT  experience  can  be  used  to  modify  the  training  requirements. 

During  OT&E,  operators  and  maintenance  personnel  gain  experience  with  the  system  and  task 
procedures  and  skills  required  to  operate  and  maintain  the  system.  This  experience  may  indicate 
required  changes  to  the  training  and  skill  needs.  The  evaluation  report  should  indicate  if  training 
and  skill  needs  should  be  changed.  Along  with  these  changes,  training  aids,  simulators,  and 
support  equipment  must  be  assessed  to  ensure  that  they  are  adequate  to  support  the  operational 
requirements. 

OT  planning  must  address  when  the  training  program  will  be  available  for  evaluation. 

The  delivery  of  all  required  training  materials  and  equipment  may  not  coincide  with  the  OT&E 
schedule.  Training  manuals  usually  are  not  developed  on  schedule,  or  they  are  in  an  early  draft 
stage.  Further,  software  changes  just  prior  to  test  may  cause  major  changes  in  system  operations, 
or  personnel  available  to  the  test  program  may  have  training  that  is  not  representative  of  that 
planned  for  the  operating  units.  Unless  OT  planning  considers  these  possibilities,  the  OT&E 
may  be  unable  to  evaluate  the  training  program. 

The  interrelationship  between  training,  documentation  (section  1.8),  and  human  factors 
(section  1.13)  must  be  recognized  during  the  OT  planning. 

Just  as  the  documentation  evaluation  is  directed  at  examining  various  important  operating  and 
maintenance  tasks,  the  ability  of  the  personnel  to  perform  the  tasks  with  the  documentation 
provided  is  dependent  on  the  training  those  personnel  have  received.  Similarly,  human  factors 
and  training  are  related.  Tasks  that  incorporate  unusual  or  complex  human  factor  aspects  may 
require  additional  or  more  extensive  training.  Combined  evaluation  of  these  areas  may  prove  to 
be  very  beneficial  and  should  be  considered  during  the  planning  for  the  OT&E.  The  adequacy  of 
the  training  for  very  complex  or  demanding  tasks  should  be  assessed  by  reviewing  the  personnel 
performance  during  the  actual  system  operation. 

Training  and  OT  tasks  should  be  correlated. 

The  correlations  between  the  tasks  included  in  the  training  and  the  tasks  performed  during  the 
operational  test  should  be  analyzed.  Tasks  should  be  identified  where  the  personnel  either  were 
not  trained  or  were  inadequately  trained.  There  also  may  be  tasks  where  training  was  determined 
to  be  unnecessary. 

Any  awkward  or  unusually  demanding  tasks  that  caused  personnel  problems  should  be 
identified. 

In  some  instances,  the  training  requirements  and  the  training  planned  are  based  upon  an 
inaccurate  view  of  the  system’s  operations  or  maintenance  activities.  OT  can  identify  unusually 
critical  tasks  that  need  to  have  the  training  re-evaluated.  This  activity  is  closely  related  to  human 
factors  (section  1.13). 
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Wartime  usage  rates  for  a  system  is  defined  as 


the  quantitative  statement  of  the  projected  manner  In  which  the  system 
Is  to  be  used  In  Its  Intended  wartime  environment 

Wanime  usage  rates  can  be  expressed  in  parameters  such  as  flying  hours  per  month,  miles  per 
day,  rounds  per  day,  or  hours  per  month.  The  full  meaning  of  these  parameters  requires  a 
definition  of  the  rate  in  relation  to  the  planned  operational  scenario.  For  example,  miles  per  day 
has  limited  meaning  unless  the  "miles"  are  characterized  by  speed,  terrain,  system  activity,  etc. 
Similarly,  sorties  per  day  is  not  a  valid  measure  unless  the  characteristics  of  the  sortie  are 
defined,  e.g.,  mission,  weapons  carried,  sortie  length,  speed. 

The  addition  of  "wartime  usage  rales"  into  the  list  of  suitability  issues  resulted  from  a  concern 
that,  in  some  instances,  the  suitability  of  systems  was  being  assessed  at  an  unrealistic  tempo 
during  the  operational  testing  period.  Wartime  usage  rates  was  added  to  emphasize  the  need  to 
have  suitability  evaluations  conducted  at  usage  rates  that  approximate  those  anticipated  during 
wartime  use.  Within  these  frequency  measures,  there  also  needs  to  be  a  description  of  the 
missions  themselves,  including  mission  duration.  Early  in  the  system's  development,  these 
mission  profiles  must  be  identified  by  the  acquisition  organization  in  conjunction  with  the  using 
and  supporting  organizations.  As  the  system  progresses  through  the  development  cycle,  it 
becomes  more  and  more  important  that  the  usage  rates  be  defined  in  greater  detail.  The  design 
must  be  made  in  the  context  of  this  usage  rate,  and  the  logistics  support  must  be  planned  in 
consideration  of  these  rates.  When  OT&E  is  planned  and  conducted,  the  wartime  usage  rates  are 
a  fundamental  part  of  determining  how  to  structure  a  test  to  determine  if  the  system  can  meet  its 
wartime  demands. 

All  of  the  operational  suitability  areas  contribute  to  the  ability  of  the  system  to  realize  the 
wartime  rates  of  usage.  The  number  of  flying  horns,  miles,  or  rounds  that  the  system  is  capable 
of  providing  is  dependent  on  its  availability  and  the  balance  between  the  logistics  demands  and 
the  logistics  resources  that  are  provided. 

PARAMETERS 

To  assess  the  ability  of  the  planned  logistics  system  to  support  a  new  weapon  system  requires 
that  the  logistics  support  be  examined  in  the  context  of  the  projected  wartime  usage  rates.  When 
a  need  is  postulated,  there  may  not  be  an  accurate  projection  of  what  the  wartime  usage  rate  is. 
These  parameters  must  be  identified  in  the  requirements  documentation.  Example  measures  are 
sorties  per  day  for  a  tactical  aircraft,  hours  per  day  for  simulators  and  some  communications 
systems,  message  units  per  time  period  for  other  communications  systems,  rounds  per  day  for 
ground  combat  systems,  etc. 

KEY  POINTS 

The  usage  parameters  should  be  identified  early  in  the  program's  development 

The  parameters  or  measures  for  wartime  usage  should  be  known  and  agreed  to  among  the 
participants  in  the  development  process  at  Milestone  L  Usually  the  parameters  for  certain 
classes  of  systems  are  relatively  easy  to  specify  and  to  reach  agreement  on.  Examples  might  be 
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miles  per  day  for  a  tank  or  ground  vehicle,  flying  hours  per  month  or  sorties  per  day  for  aircraft, 
hours  per  month  for  surveillance  systems,  or  rounds  per  day  for  an  artillery  piece. 

Usage  parameters  must  be  fully  defined. 

The  wartime  usage  rate  must  have  full  definition  of  the  wartime  mission  or  scenario  for  the  rate 
to  have  meaning.  Message  rate  per  day  may  be  an  acceptable  usage  rate  for  a  communications 
system,  but  the  content  and  complexity  of  the  messages  must  be  defined  to  give  the  statement 
meaning.  These  definitions  can  be  a  composite  of  types,  i.e.,  “x”  percent  of  these  message,  “y” 
percent  of  those,  etc.  These  defined  usage  rates  will  be  documented  in  the  Operational  Mode 
Summary,  Mission  Profile,  or  similar  documents. 

Usage  rates  should  be  developed  with  the  new  system’s  capabilities  in  mind. 

If  the  system's  usage  rates  are  based  upon  predecessor  systems’  experience,  these  rates  should  be 
analyzed  by  the  intended  users  to  assure  that  they  still  are  valid.  In  many  cases,  new  systems 
with  improved  capabilities  have  been  found  to  have  very  different  use  patterns  once  they  are 
placed  with  the  operating  units.  Examples  are  surveillance  systems  that  were  used  infrequently 
until  they  were  replaced  by  higher  capability  systems.  The  using  organizations  were  so  pleased 
with  the  improved  systems  that  the  usage  increased  significantly  to  almost  continuous  use. 

The  operating  tempo  during  OT  should  be  developed  from  the  planned  usage  rates. 

By  Milestone  II,  the  value  for  the  wartime  usage  rates  should  be  known  and  documented  in  an 
Operational  Mode  Summary,  or  similar  document  These  rates  also  should  be  included  in  the 
Milestone  II  TEMP,  by  reference.  The  developing  agencies  should  understand  the  usage  rates. 
They  also  should  be  used  by  the  agencies  performing  the  logistics  planning,  and  should  be  part 
of  the  requirements  used  in  the  planning  of  the  OT&E  program. 

The  OT  may  be  incapable  of  directly  demonstrating  the  wartime  usage  rates. 

The  planned  OT  should  test  the  wartime  usage  as  specified  in  the  Operational  Mode  Summary, 
or  mission  profile.  Any  attempt  to  avoid  testing  under  typical  combat  conditions/wartime  usage 
rates  should  be  examined  carefully,  as  such  unrealistic  operation  could  significantly  alter  the 
suitability  performance  seen  during  testing  and  serve  as  a  severe  test  limitation.  If  the  planned 
OT  is  incapable  of  testing  the  system  at  the  high  level  of  wartime  usage,  then  modeling  and 
simulation  should  be  used  to  project  the  system's  capability  at  rates  higher  than  those  seen  during 
the  operational  testing.  This  modeling  and  simulation  requirement  should  be  identified  by  the 
OTA  in  the  Milestone  n  TEMP,  and  planning  initiated  to  develop  and  validate  the  required  mod¬ 
els  and  simulations. 

Some  evaluation  must  be  made  of  the  system's  capability  to  perform  at  the  planned 
wartime  usage  rates. 

The  capability  of  the  system  to  perform  at  wartime  usage  rates  should  be  assessed  in  the  OT&E 
prior  to  Milestone  HI,  or  IIIA  if  there  is  one.  If  there  is  doubt  about  the  ability  of  the  logistics 
system  to  support  the  system  at  this  rate  or  the  ability  of  the  system  to  perform  at  this  rate,  then 
the  test  report  should  highlight  these  conclusions.  If  there  is  a  perceived  limitation  on  the  system 
usage  rate,  this  limitation  should  be  highlighted  in  the  OT&E  report.  As  an  example,  an  aircraft 
system  was  incapable  of  achieving  its  squadron  level  sortie  rate  because  of  the  demand  placed  on 
the  second  level  test  equipment  and  the  number  of  test  stations  planned  for  each  squadron.  The 
highlighting  of  this  deficiency  resulted  in  a  re-evaluation  of  the  number  of  test  stations. 
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Safety  Is  defined  as 

freedom  from  those  conditions  that  can  cause  death,  Injury, 
occupational  Illness,  damage  to  or  loss  of  equipment  or  property,  or 
damage  to  the  environment. 

Safety  is  an  essential  and  integral  part  of  assessing  a  system  in  an  operational  environment  It 
addresses  any  potential  hazards  that  the  system  (both  hardware  and  software)  poses  to  personnel, 
or  other  systems  or  equipment  Safety  usually  is  evaluated  by  observing  the  system's  use  and 
maintenance  while  performing  other  portions  of  the  operational  testing.  Because  the  safety 
assessment  is  a  byproduct  of  this  testing,  it  is  important  to  ensure  that  safety  aspects  of  the 
system's  use  arc  not  overlooked  as  the  principal  attention  is  focused  on  other  aspects  of  the 
system's  testing.  Since  the  OT&E  may  be  the  first  instance  where  the  system  will  be  operated 
and  used  in  its  planned  environment,  this  also  may  be  the  first  instance  where  it  will  be  possible 
to  observe  any  potential  safety  problems. 

The  acquisition  organization  usually  will  conduct  system  safety  programs  on  more  complex 
systems'  The  system  safety  program  uses  engineering  and  management  techniques  to  identify 
and  eliminate  potential  hazards  and  reduce  associated  risks. 

PARAMETERS 

During  some  operational  tests,  the  OTA  may  use  the  categories  and  hazard  levels  from  the 
system  safety  program  as  a  way  of  identifying  the  results  of  the  OT  from  a  safety  standpoint 
The  parameters  for  system  safety  relate  to  the  number  of  hazards  in  specified  categories  and  the 
projected  frequency  of  exposure-  to  these  classes  of  hazards. 

Number  of  Hazards  by  Category:  MIL-STD-882  has  a  series  of  four  hazard  categories,  with 
Category  I,  Catastrophic,  being  the  most  serious.  The  categories  arc: 


Description 

.Cflieggiy 

MishaD  Definition 

Catastrophic 

I 

Death,  or  system  loss 

Critical 

n 

Severe  injury,  severe  occupational 
illness,  or  major  system  damage 

Marginal 

m 

Minor  injury,  minor  occupational 
illness,  or  minor  system  damage 

Negligible 

IV 

Less  than  minor  injury, 
occupational  illness,  or  system 
damage 

For  most  systems,  the  objective  is  to  eliminate  all  Category  I  and  II  hazards.  Also,  the  hazards 
that  are  identified  during  operational  testing  should  be  categorized,  if  possible. 
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Hazard  Probability:  If  possible,  any  observed  hazards  should  be  identified  by  the  probability 
levels  that  are  used  in  system  safety  programs.  This  designation  may  aid  in  the  investigation  and 
resolution  of  the  hazards.  The  hazard  probability  levels  (contained  in  MIL-STD-882)  are: 


Level 

Probability 

Definition 

A 

Frequent 

Likely  to  occur  frequently 

B 

Probable 

Will  occur  several  times  in  the 
life  of  the  item 

C 

Occasional 

Likely  to  occur  sometime  in  life 
of  an  item 

D 

Remote 

Unlikely,  but  possible  to  occur  in 
life  of  an  item 

E 

Improbable 

So  unlikely  that  it  can  be  assumed 
occurrence  may  not  be  experienced 

Kt¥  pOiNlS 

Operational  testing  provides  an  opportunity  to  observe  the  system  operated  and  supported 
by  personnel  having  the  expected  skill  levels. 

Prior  to  operational  testing,  the  personnel  who  operate  and  maintain  the  system  probably  will 
have  higher  skill  and  experience  levels  than  will  the  planned  operational  personnel.  Therefore, 
the  OT  is  the  first  opportunity  to  observe  the  system  in  the  hands  of  personnel  with  the  projected 
levels  of  experience  and  skills.  This  first  observation  may  indicate  potential  safety  problems  that 
were  not  observed  in  the  earlier  testing.  Test  planning  should  focus  attention  on  detecting  and 
documenting  any  new  or  unexpected  hazards  to  personnel.  This  observation  is  closely  related  to 
human  factors  assessment  (see  section  1.13). 

Observers  of  operational  tests  should  be  sensitive  to  any  potential  for  significant  hazards. 

All  personnel  who  are  involved  in  or  associated  with  the  conduct  of  the  operational  testing  have 
a  responsibility  to  identity  any  potential  hazard,  and  cause  the  test  to  be  stopped  if  a  hazard  in 
Categories  I  or  It  is  perceived.  In  most  cases,  operational  testing  should  be  conducted  without 
outside  interference,  but  safety  is  an  exception. 

Safety  testing  should  consider  the  operating  environment  of  the  system. 

Any  safety-oriented  testing  or  assessment  should  consider  the  entire  expected  range  of 
environments.  Some  safety  features  may  be  very  effective  in  good  weather  on  a  clear  day. 
Hazards  may  be  clearly  seen  and  easily  avoided.  In  poor  lighting  or  in  bad  weather,  poor 
visibility  may  result  in  unexpected  hazardous  conditions. 

Software  faults  can  result  in  unexpected  hazards. 

Faults  observed  in  the  software  should  be  evaluated  for  potential  contribution  to  hazardous 
conditions.  As  an  example,  for  aircraft  systems,  software  faults  can  impact  flight  safety. 
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i.is  human  fagtarspu  V. 

Human  Factors  is  defined  as 

those  elements  of  system  operation  and  maintenance  which  Influence 
the  efficiency  with  which  people  can  use  systems  to  accomplish  the  op¬ 
erational  mission.  The  Important  elements  of  human  factors  are  the 
equipment  (e.g.,  arrangement  of  controls  and  displays),  the  work  envi¬ 
ronment  (e.g.,  room  layout,  noise  level,  temperature,  lighting,  etc.),  the 
task  (e.g.,  length  and  complexity  of  operating  procedures),  and  person¬ 
nel  (e.g.,  capabilities  of  operators  and  maintainors). 

This  suitability  element  addresses  the  compatibility  among  system  hardware  and  software 
elements  and  the  human  elements.  It  is  intended  to  identify  system  performance  problems, 
human  task  performance  problems,  and  hazards  to  personnel  under  realistic  conditions  of  combat 
use.  While  there  is  a  close  alliance  between  the  human  factors  examination  and  the  examination 
of  manpower  supportability  (see  section  1.9),  training  (see  section  1.10),  and  safety  (see  section 
1.12),  human  factors  is  focused  more  on  the  hardware  and  software  elements  of  the  system.  It  is 
more  of  an  evaluation  of  the  system  itself,  what  the  system  requires  of  the  people  who  operate 
and  maintain  it,  and  how  the  system  fits  into  the  relationship  with  the  people  who  are  going  to 
operate  and  maintain  it. 

Evaluation  of  the  human  factors  aspects  of  a  new  system  includes  all  of  the  interfaces  between 
personnel  and  the  hardware  and  software.  It  also  includes  the  interfaces  with  both  the  system 
operators  and  the  system  maintenance  personnel. 

The  human  factors  considerations  include  compatibility  of  the  man-machine  interface. 
Considerations  in  this  area  comprise  information  displays  (machine  feedback),  symbology 
(standard  versus  unique),  operator  controls,  personnel  comfort  and  convenience,  portability  of 
the  equipment  (bulk,  weight,  load  distribution,  straps,  handles,  etc.),  accessibility  for  operation 
and  maintenance,  physical  workload  demands,  mental  workload/information  processing 
demands,  compatibility  with  task  characteristics,  task  environment,  etc. 

Traditional  human  factors  testing  also  may  address  ergonomic  (e.g.,  can  the  operator  see  or  reach 
the  indicator  or  control)  considerations,  although  this  area  normally  is  part  of  the  developmental 
testing.  The  OT&E  should  address  the  operator’s  effectiveness  and  efficiency  in  the  performance 
of  assigned  tasks. 

PARAMETERS 

Human  factors  is  evaluated  qualitatively  using  checklists  that  focus  attention  on  the  human  fac¬ 
tors  aspects  of  the  system.  It  also  can  be  evaluated  quantitatively  by  performing  timed  tasks. 
Data  to  assist  in  the  evaluation  may  be  collected  on  task  times,  response  times,  error  rates,  accu¬ 
racy,  etc.  Interviews,  questionnaires,  and  debriefings  of  operators  and  maintenance  personnel 
can  be  used  to  gather  data  on  impressions  of  displays,  man-machine  interface,  accessibility,  port¬ 
ability,  task  environment,  task  difficulty,  unnecessary  steps,  work  space,  personnel  fatigue,  etc. 
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KEY  POINTS 

Human  factors  should  address  both  operators  and  maintenance  personnel. 

Human  factors  evaluation  should  address  the  ability  of  the  operators  to  use  and  to  control  the 
functioning  of  the  system.  It  also  should  ensure  that  the  support  personnel  have  the  access  and 
the  physical  capability  to  perform  the  required  maintenance. 

The  software  interface  with  personnel  should  be  assessed. 

As  systems  become  more  software  intensive,  there  is  a  need  to  evaluate  the  interface  of  the 
software  with  the  operators  and/or  maintenance  personnel.  How  does  the  software  present 
information?  Is  it  clear,  or  can  there  be  misinterpretation?  Is  there  consistency  among  the 
various  software  displays  so  the  operator:,  will  have  an  acceptable  learning  curve  as  the  system  is 
used?  Is  software  designed  with  safeguards,  to  the  extent  possible,  against  system  failures  due  to 
incorrect  key  entries?  Some  OTAs  use  checklists  and  questionnaires  to  examine  the  "usability” 
of  the  software. 

Physical  demands  on  personnel  should  be  assessed. 

Consideration  needs  to  be  given  to  the  manner  in  which  the  system  is  to  be  used  under  combat 
conditions.  Are  the  personnel  likely  to  be  operating  or  maintaining  the  system  while  wearing 
protective  clothing  against  the  weather  or  against  chemical,  nuclear,  or  biological  attack?  How 
long  will  they  have  to  operate  under  these  conditions  for  the  system  to  perform  its  mission(s)? 
Does  the  system  require  relatively  k»g  periods  of  concentration,  or  exertion?  Some  soldier-fired 
systems  require  the  sights  to  be  kept  on  the  target  for  a  long  period  of  time.  Is  this  period 
realistic  in  terms  of  human  capability  for  the  average  person? 

The  employment  of  new  or  advanced  display  techniques  should  be  identified. 

When  new  or  advanced  display  techniques  are  used  in  the  operator’s  station  or  in  maintenance 
equipment,  there  may  be  significant  questions  about  how  these  changes  will  be  accepted  and 
integrated  into  the  operation  of  the  line  operating  unit.  A  COI  should  be  identified  ?nd  the  OT 
should  ensure  that  any  potential  problems  are  e-ami:  jd  in  adequate  detail. 

Human  factor  conditions  should  consider  the  entire  operating  environment 

The  examination  of  the  man-machine  interface  needs  to  consider  that  the  system  will  be  operated 
under  a  wide  rage  of  conditions.  If  needed,  can  the  system  be  used  effectively  with  arctic 
clothing,  CBR  protective  clothing,  etc.?  Can  it  be  effectively  used  in  poor  weather,  or  with 
limited  lighting?  In  one  example,  the  OTA  took  user  troops  to  the  contractor's  facility  where 
they  found  that  the  systems  could  not  be  assembled  by  the  planned  field  personnel.  The/  also 
discovered  that  tactical  personnel  could  not  enter  some  of  the  shelters  with  packs  and  weapons, 
forcing  them  to  leave  these  items  outside  in  potentially  hostile  or  contaminated  environments. 

Combat  stress  conditions  can  affect  tbe  ability  of  personnel  to  operate  or  maintain  tbe 
system,  and  should  be  evaluated. 

Concerns  for  the  ability  of  personnel  to  use  and  repair  tire  system  under  combat  stress  conditions 
must  be  addressed.  Under  very  stressful  conditions,  often  only  the  simplest  tasks  succeed. 
Simulating  the  combat  stress  factor  is  very  difficult  The  number  and  frequency  of  different 
tasks  should  be  analyzed  and  documented  in  the  operational  test  plan.  The  plan  should  include 
the  measures  that  will  be  used  to  gain  insight  into  the  reactions  of  the  people  to  the  stress  of  the 
proposed  environment  and  their  ability  to  perform  as  required. 
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Chapter  2 

OTHER  OPERATIONAL  SUITABILITY  ISSUES 


In  addition  to  the  suitability  elements  that  are  enumerated  in  the  definition  of  operational 
suitability  and  are  discussed  in  Chapter  1,  there  are  additional  topics  that  are  essential  to  a  dis¬ 
cussion  of  operational  suitability  in  OT&E.  These  topics  need  emphasis  and  understanding  for 
operational  suitability  to  be  effectively  evaluated  during  OT&E.  Among  these  topics  are 
suitability  modeling  and  simulation,  integrated  diagnostics,  environmental  factors,  electromag¬ 
netic  environmental  effects  (E3),  and  software  supportability.  The  following  sections  discuss 
these  topics. 
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modeling  and  simulation 


The  DoD  Is  in  the  process  of  issuing  expanded  guidance  on  the  development, 
validation,  and  use  of  modeling  and  simulation  (M&S)  in  the  acquisition  process. 
In  January  1989,  the  Director  of  Operational  Test  and  Evaluation  (DOT&E)  issued 
the  "DOT&E  Policy  for  the  Application  of  Modeling  and  Simulation  in  Support  of 
Operational  Test  and  Evaluation." 

A  model  is  defined  as 

a  representation  of  an  actual  or  conceptual  system  that  Involves 
mathematics,  logical  expressions,  or  computer  simulations  that  can  be 
used  to  predict  how  the  system  might  perform  or  survive  under  various 
conditions  or  In  a  range  of  hostile  environments. 

Simulation  is  defined  as 

a  method  for  Implementing  a  model.  It  Is  the  process  of  conducting 
experiments  with  a  model  for  the  purpose  of  understanding  the  behavior 
of  the  system  modeled  under  selected  conditions  or  of  evaluating 
various  strategies  for  the  operation  of  the  system  within  the  limits 
Imposed  by  developmental  or  opemtlonal  criteria. 

There  are  several  different  types  of  simulations,  including  those  that  use  analog  or  digital 
devices,  laboratory  models,  or  "test-bed"  sites. 

The  use  of  properly  validated  M&S  is  strongly  encouraged  during  the  early  phases  of  a  program 
to  assess  those  areas  that  cannot  be  directly  observed  through  testing.  The  use  of  M&S  is  not  a 
substitute  for  actual  testing;  however,  it  can  provide  early  projections  and  reduce  test  costs  by 
supplementing  actual  test  data. 

The  use  of  modeling  and  simulation  in  the  operational  suitability  area  can  provide  a  number  of 
benefits.  M&S  can  be  used  to  focus  limited  test  resources  by  identifying  the  critical  elements  in 
a  logistics  support  system,  e.g.,  the  choke  points  for  the  flow  of  the  support  resources.  M&S  also 
can  be  used  to  translate  the  rate  of  use  in  the  test  scenario  to  the  wartime  usage  rate.  If,  for  ex¬ 
ample,  test  aircraft  are  flying  only  one  or  two  sorties  per  day,  the  "load"  on  the  support  resources 
is  significantly  different  than  if  a  higher,  wartime  sortie  rate  was  being  flown.  M&S  can  aid  in 
assessing  the  impact  of  these  differences.  M&S  also  may  be  used  to  evaluate  elements  of  the 
support  system  that  are  not  present  at  the  test  site.  For  example,  if  the  second-level  maintenance 
capability  (test  equipment,  facilities,  etc.)  is  not  available,  then  a  properly  constructed  and 
validated  model  can  be  used  to  provide  insight  into  the  ability  of  the  planned  second-level 
maintenance  facility  to  support  the  system. 

PARAMETERS 

The  key  parameters  in  M&S  are  the  assumptions  and  ground  rules  used  in  inputting  data.  The 
output  can  only  be  valid  if  valid  assumptions  and  ground  rules  have  been  used. 
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KEY  POINTS 

Plans  for  M&S  should  be  evaluated  for  potential  credibility  of  the  results. 

The  credibility  of  the  results  of  M&S  is  a  judgment  formed  from  the  composite  of  impressions  of 
the  inputs,  processes,  outputs,  conclusions,  the  persons  or  agencies  involved,  and  the  strength  of 
the  evidence  presented.  Appendix  B  of  the  "DOT&E  Policy  for  the  Application  of  Modeling  and 
Simulation  in  Support  of  Operational  Test  and  Evaluation''  provides  a  series  of  questions  to 
assist  in  assessing  M&S  results’  credibility.  These  questions  provide  a  good  outline  for 
examining  M&S  activities. 

All  of  the  models  planned  for  use  on  the  OT&E  program  should  be  accredited  for  the 
purpose. 

Accreditation  is  defined  as  the  process  of  certifying  that  a  computer  model  has  achieved  an  es¬ 
tablished  standard  such  that  it  can  be  applied  for  a  specific  purpose.  This  means  that  mana°e- 
ment  has  examined  the  model  and  based  upon  experience  and  expert  judgment,  has  declared  that 
the  model  is  adequate  for  its  intended  use. 

Detailed  definitions  of  planned  operating  and  support  scenarios  are  essential  for  a  valid 
M&S  effort 

In  many  cases,  the  detailed  definition  that  is  needed  for  M&S  is  beyond  that  existing  in  program 
documentation.  This  is  particularly  true  in  the  suitability  area,  where  the  maimrnan™-  and 
supply  concepts  to  be  used  must  be  defined  in  detail.  There  is  a  potential  for  the  M&S  results  to 
be  driven  by  some  of  die  necessary  assumptions  rather  than  by  the  system  charw-trri«i/-«:  On  the 
other  hand,  if  the  responsible  personnel  and  organizations  are  requested  to  provide  the  required 
(retail,  and  the  support  planning  is  thought  through,  then  other  organizations  within  the  program 
also  will  benefit 

The  latest  program  information  must  be  incorporated  into  the  M&S  activity. 

Many  modeling  efforts  lack  current  information.  Program  conditions  may  change.  The  system 
design  may  be  revised,  or  new  threat  information  received.  In  each  case,  the  earlier  model  may 
be  invalidated.  Assuring  that  die  modeling  results  reflea  the  best  and  most  current  information 
available  is  an  important  consideration.  Procedures  must  be  established  to  assure  that  current 
information  is  provided  to  those  doing  the  modeling  and  evaluation  of  the  simulation  results. 

Defined  plans  for  the  use  of  M&S  should  be  presented  in  the  TEMP. 


The  TEMP  should  indicate  any  plans  for  the  use  of  suitability  modeling  and  to 

complement  or  supplement  the  operational  testing.  The  models  to  be  used  should  be  iHcntifi-tt 
and  plans  for  then  validation  described.  M&S  should  not  be  used  in  place  of  actual  testing. 

The  TEMP  and  other  test  documentation  should  include  a  discussion  of  the  rationale  for 
the  selection  of  the  specific  models  that  are  planned  for  suitability  analysis. 

Models  arc  used  for  many  of  the  assessments  for  suitability.  The  Services  should  list  the  models 
and  discuss  their  advantages/disadvantages  in  the  TEMP  or  other  test  documentation  so  that  an 
evaluation  can  be  made  as  to  the  utility  of  the  model.  The  DOT&E  evaluator  must  be  able  to  as¬ 
sess  the  validity  of  the  selected  models  and  of  the  OTA’s  assessment  results  ftom  the  model’s 
use. 
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2,2  INTIS©  RATED  DIAGNOSTIC© 


Diagnostics  is  defined  in  the  OTAs  Multi-Service  Testing  MOA  as 

the  ability  of  integrated  diagnostics  (automated,  semi-automated,  and 
manual  techniques  taken  as  a  whole)  to  fault-detect  and  fault-isolate  In  a 
timely  manner. 

Integrated  Diagnostics  is  defined  as 

a  structured  process,  which  maximizes  the  effectiveness  of  diagnostics 
by  Integrating  pertinent  elements,  such  as  testability,  automatic  and 
manual  testing,  training,  maintenance  aiding,  and  technical  Information 
that  will  satisfy  weapon  system  peacetime  and  combat  mission  require¬ 
ments  and  enable  critical  failures  to  be  fixed  with  minimum  loss  of  oper¬ 
ational  availability. 

The  purpose  of  integrated  diagnostics  is  to  provide  a  cost-effective  capability  to  detect  and  un¬ 
ambiguously  isolate  all  faults  known  or  expected  to  occur  in  weapons  systems  and  equipment  in 
order  to  satisfy  weapon  system  mission  requirements.  In  wartime,  this  becomes  extremely  sig¬ 
nificant  in  that  it  is  imperative  that  critical  failures  be  found  and  fixed  quickly  to  support  combat 
turn-around  times,  which  can  equalize  battles  against  numerically  superior  forces 

The  term  “Diagnostics”  often  is  used  as  a  general  term  to  cover  all  means  of  determining  that  a 
system  fault  has  occurred,  and  the  means  to  determine  where  the  fault  is  and  to  isolate  it  to  a 
portion  of  the  system  that  can  be  repaired  or  replaced.  There  are  many  other  terms  that  are  used 
in  this  area,  including  Built-In  Test  (BIT),  Built-In  Test  Equipment  (BITE),  Built-In  Test  and 
Fault  Isolation  Test  (BIT/FIT),  and  Automatic  Test  Equipment  (ATE). 

The  key  to  integrated  diagnostics  is  die  successful  consideration  and  integration  of  the  functions 
of  detection,  isolation,  verification,  recovery,  recording,  and  repotting,  in  a  comprehensive  and 
cohesive  fashion,  with  the  operator  and  with  support  functions  that  may  be  automatically, 
semiautomatically  and/or  manually  controlled. 


PARAMETERS 

Two  parameters  are  listed  in  the  OTAs  MOA  for  diagnostics  use  in  multi-Scrvice  OT&E  test 
programs.  They  are: 

Percent  of  Correct  Detection  given  that  a  fault  has  occurred  (P^j), 


Pcd  - 


The  number  of  correct  detections 
The  total  number  of  confirmed  faults 


Mean  Tune  to  Fault  Locate  (MTTFL), 


MTTFL= 


The  amount  of  time  required  to  locate  faults 
The  total  number  of  faults 
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These  parameters  apply  to  a  specified  level  of  maintenance  and  therefore  may  be  applicable  to 
each  level  of  maintenance. 

The  ability  of  an  automated  diagnostics  system  to  isolate  faults  also  may  be  measured.  One  pa¬ 
rameter  for  this  characteristics  is  percent  fault  isolation. 


Percent 

Fault 

Isolation 


Number  of  fault  isolations  in  which 
automated  diagnostics  effectively  contributed 
Number  of  confirmed  failures 
detected  via  all  methods 


Another  important  measure  of  the  capability  of  the  diagnostics  system  is  to  identify  how 
frequently  the  automated  diagnostics  indicates  that  a  fault  exists  when  in  fact  the  system  is 
functional.  This  area  is  particularly  troublesome,  since  the  system  false  alarms  may  be  either 
improper  indications  of  faults  that  do  not  exist,  or  faults  that  did  exist  but  were  transient  in 
nature.  Identifying  which  situation  exists  is  most  difficult  for  complex  systems.  One  of  the  more 
common  parameters  for  false  alarms  measurement  is  BIT  false  alarm  rate  (expressed  as  a  per¬ 
centage). 

Pfwnr  Number  of  BIT  indications  not 
BITFalse  =  resulting  in  maintenance  actions  x 
Alarm  Total  number  of  BIT  indications 


Another  important  aspect  of  integrated  diagnostics  is  the  compatibility  of  the  various  levels  of 
testing.  The  faults  that  arc  detected  by  the  automatic  built-in  test  must  be  verified  by  a  more 
thorough  diagnostics  system  than  is  available  to  maintenance  personnel.  The  verification  of  a 
fault  after  removal  of  the  equipment  from  the  system  platform  is  sometimes  exasperating  be¬ 
cause  the  maintenance  test  station  cannot  duplicate  the  operational  environment  (vibration, 
temperature,  etc.)  that  was  present  in  the  instance  of  the  initial  fault  reporting.  Once  the  failed 
item  is  removed,  the  test  equipment  at  the  next  level  of  maintenance  must  be  able  to  identify  the 
same  fault  Two  parameters  are  commonly  used  in  this  area:  the  “cannot  duplicate”  (CND)  rate 
and  the  “retest  okay”  rate.  Both  are  expressed  as  a  percentage. 

Percent  ,  Number  of  faults  that  could  not  be 
Cannot  _  duplicated  by  later  maintenance  actions 
Duplicate  Total  number  of  faults  reported  X 

Percent  Number  of  faults  that  could  not  be 

Retest  _  ouplicated  at  the  next  level  of  maintenance  ]nft 

Okay  Total  number  of  faults  reported  X 

These  last  three  equations  are  based  on  a  percentage  of  BIT  indications.  Improved  parameters, 
used  on  some  new  systems,  use  a  measure  of  life  units  (hours,  miles,  sorties)  in  the  denominator. 
This  results  in  parameters  such  as  number  of  BIT  false  ianns  per  sortie  or  per  hour. 

In  addition  to  the  parameters  discussed  here,  each  of  the  Services  employs  numerous  other  gen¬ 
eral  and  unique  parameters  in  their  respective  programs.  These  include  parameters  that  relate  to 
tiie  manual,  as  well  as  automatic  and  semi-automanc  aspects  of  integrated  diagnostics. 
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The  approach  to  system  diagnostics  should  be  discussed  in  the  early  system  planning 
documents. 

These  discussions  provide  a  basis  for  relating  the  diagnostics  requirements  to  other  system 
parameters  such  as  reliability,  maintainability,  and  availability. 

All  aspects  of  the  integrated  diagnostics  function  must  be  planned  for. 

All  integrated  diagnostics  test  items  required  in  the  support  of  the  weapon  system  must  be 
planned  for.  Focusing  on  the  exotic  on-board  built-in  test  features  can  lead  to  only  minimal 
planning  for  the  less  exotic  support  functions  such  as  automatic  test  equipment,  test  program 
sets,  technical  manuals,  training,  and  required  skill  levels  of  personnel. 

The  program  manager  should  have  firm  diagnostics  requirements  established  before 
Milestone  U. 

Diagnostics  requirements  should  be  the  result  of  analysis  that  allocates  the  diagnostics 
requirements  across  the  various  alternatives,  i.e.,  automated  systems,  semi-automated  systems, 
test  equipment,  and  manual  troubleshooting.  The  initial  operational  testing  should  provide 
insight  into  the  system's  capability  versus  these  allocated  levels  of  diagnostics.  The  total 
diagnostics  capability  may  meet  the  threshold,  but  if  some  elements  are  far  from  what  is 
required,  the  system  still  may  be  not  suitable. 

Diagnostics  short-falls  should  be  evaluated  by  the  OTA  as  to  the  total  impact  on  the  system 
and  its  support  resources. 

When  operational  test  results  arc  presented  at  the  Milestone  III  decision,  the  evaluation  of 
diagnostics  capability  should  discuss  the  relative  effect  of  the  diagnostics  performance  on  the 
reliability,  maintainability,  and  availability  of  the  system.  For  example,  what  is  the  impact  on 
the  system  availability  if  the  is  less  than  the  threshold? 

Diagnostics  short-falls  may  be  obscured  by  activities  in  other  suitability  areas. 

While  many  current  systems  have  failed  to  realize  the  level  of  required  diagnostics,  these 
deficiencies  have  not  always  been  corrected  prior  to  the  system  being  fielded,  but  have  been 
offset  by  changes  in  other  parts  of  the  support  system.  For  example,  the  inability  of  automated 
systems  to  perform  the  level  of  fault  isolation  that  was  expected  may  lead  to  an  expanded  use  of 
manual  troubleshooting  on  some  portion  of  the  system.  The  allocation  of  the  diagnostic  task  to 
the  various  alternative  methods  must  be  assessed  as  part  of  OT&E.  If  the  allocation  cannot  be 
realized,  then  the  impact  of  a  reallocation  must  be  assessed,  along  with  the  penalties  in  cost  or 
readiness  that  an  adjustment  of  the  allocation  conveys. 
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Indications  of  poor  performance  of  the  system  on-board  diagnostics  early  in  the  program 
should  be  followed  closely,  as  lack  of  diagnostics  performance  can  lead  to  major  suitability 
problems. 

Failure  of  a  system  to  perform  the  planned-for  on-board  diagnostics  can  have  serious  impacts  on 
the  support  structure.  If  it  becomes  necessary  to  perform  those  functions  off-platform,  adequate 
types  and  quantities  of  support  equipment  may  not  be  planned  for  and  the  planned  training  and 
personnel  skill  levels  may  not  be  able  to  absorb  the  required  additional  burden. 

A  common  problem  with  diagnostics  is  its  immaturity  at  the  early  stages  of  operational 
testing. 

When  the  system  is  tested  in  its  early  stages,  the  diagnostics  may  be  less  capable  than  desired,  or 
result  in  numerous  false  fault  indications.  Resulting  deficiencies  are  labeled  as  the  result  of 
immaturity  and,  more  often  than  not,  it  is  projected  that  the  mature  system  will  not  have  these 
problems.  The  maturing  of  a  diagnostics  system  is  a  difficult  and  demanding  task.  Revising  the 
diagnostics  approach  to  a  system  design  that  is  fairly  fixed  generally  will  not  yield  significant 
improvement  The  expectation  of  significant  improvement  must  be  accompanied  by  a 
maturation  program  that  has  the  proper  resources  to  do  the  job  and  the  operational  testing  to 
verify  the  results. 

The  automated  diagnostics  capability  of  a  system  usually  improves  as  the  system's  design 
matures. 

The  automated  diagnostics  capability  of  a  system  is  one  of  the  system  features  that  usually  is  not 
completed  with  the  initial  design.  The  testing  and  operation  of  the  system  provides  additional 
insights  into  both  the  system's  performance  and  the  potential  failure  modes  and  effects.  Such  in¬ 
formation  that  results  from  development  activities  should  be  used  to  improve  the  diagnostics 
capability.  The  impact  of  this  situation  on  OT&E  is  that  the  maturity  of  the  diagnostics  at  the 
time  of  operational  testing  needs  to  be  evaluated  prior  to  the  testing,  and  the  test  results  need  to 
be  evaluated  in  light  of  system  maturity. 

Poor  diagnostics  performance  can  have  serious  effects  on  the  system's  suitability. 

If  the  system  has  poor  diagnostics  performance  during  operational  testing,  the  impact  may  be  felt 
in  a  number  of  areas.  The  operators  and  maintenance  personnel  may  loose  confidence  in  the 
diagnostics  system.  If  incorrect  system  status  is  frequently  displayed  to  the  operators,  they  will 
be  unable  to  rely  on  the  system  displays  (this  can  be  a  particular  problem  with  protection 
systems,  such  as  electronic  warfare  systems).  Similarly,  if  the  maintenance  personnel  perceive 
that  the  automatic  diagnostics  system  is  not  accurate  much  of  the  time,  they  will  have  to  resort  to 
other  means  to  maintain  the  system.  This  may  result  in  unrealistic  data  from  the  OT  or  unneces¬ 
sary  demands  being  placed  on  the  supply  system,  or  it  may  cause  addition  requirements  in 
documentation,  training,  or  other  logistics  areas  for  the  operational  system. 
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The  "operational  environment"  Is  a  critical  factor  to  the  operational  suitability  of 
a  system.  The  ability  of  the  DoD  operational  testing  organizations  to  determine 
the  operational  suitability  during  OT&E  is  dependent  on  how  the  test  environment 
compares  to  the  operating  environment  The  "operational  environment"  is 
composed  of  many  individual,  distinct  and  almost  unrelated  areas.  Often,  each 
area  must  be  addressed  separately  to  ensure  adequate,  deliberate  consideration. 

In  the  context  of  this  guide,  the  definition  of  "environment"  maintains  a 
breed  scope  and  Includes  the  weather:  vegetation;  terrain  (land  or 
water);  acoustic;  electrical/electronic;  Illumination;  chemical,  biological, 
radiation  (CBR);  and  battlefield  conditions.  There  are  two  major 
categories  of  environment;  natural  and  man-made. 

The  framework  for  discussing  environments  is  shown  in  Table  2-1. 

Any  environmental  condition  may  have  an  impact  on  the  system's  performance  and  the  ability  to 
properly  use  the  system  in  the  intended  combat  or  wartime  environments.  The  word 
"environment"  also  may  be  modified  by  an  appropriate  explanatory  adjective,  e.g.,  combat 
environment,  human  environment,  vibrational  (mechanical)  environment,  and  so  forth.  Taken 
together,  these  encompass  what  might  be  considered  the  "operational  environment."  Care  must 
be  exercised  in  the  preparation  of  OT&E  documents  to  ensure  that  the  writer  and  the  reader 
similarly  interpret  the  discussion  of  environment 

When  an  operational  need  is  stated  for  a  new  system,  it  is  necessary  to  state  what  the  conditions 
for  use  will  be.  These  use  conditions  include  the  environmental  factors  that  bear  on  the  utility  of 
the  system.  Any  system  limitation  that  is  postulated  due  to  environmental  factors  or  conditions 
should  be  identified  by  the  user  or  user  representatives  who  are  responsible  for  developing  the 
system  level  requirements.  These  limitations  also  should  be  identified  for  examination  as  part  of 
the  OT&E.  The  OT&E  should  be  planned  in  such  a  manner  so  as  to  determine  if  the  level  of 
limitation  is  as  expected,  or  if  it  is  more  severe  than  estimated.  Testing  also  should  determine  if 
the  limitation  affects  the  system  in  a  manner  other  than  what  was  predicted. 

The  operational  requirement  should  state  the  general  operating  environment  and  indicate  if  the 
requiremer.  includes  any  limitation  to  operational  use  due  to  the  environment.  That  is,  does  the 
system  comprise  elements  that  are  sensitive  to  environmental  conditions  (e.g.,  rain,  fog)  and  also 
battlefield  conditions  (e.g.,  smoke,  dust)? 
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Table  2-1  A  Framework  for  Discussing  Environments 


ENVIRONMENT 

NATURAL 

(EXAMPLES) 

MAN-MADE 

(EXAMPLES) 

WEATHER 

Rain,  Snow,  Winds, 
Sea  State,  Fog 

— 

VEGETATION 

Grass,  Shrubs,  Trees 

— 

TERRAIN 

Swamp,  Desert, 
Mountains,  lee. 
Plains,  Water,  Soil 

*  Moats,  Fox  Holes, 

Tank  Traps,  Roads, 
Urban  Features 

ACOUSTIC 

Thunder,  Rain,  Fish, 
Whales,  Waves 

* 

Decoys,  Ships 

ELECTRICAL/ 

ELECTRONIC 

Lightning,  Solar 
Flares,  Ionospheric 
Disturbances 

* 

Jamming,  EMP 

ILLUMINATION 

Sun,  Moon,  Eclipse 

* 

Flares,  Searchlights 

CBR 

Space  Radiation, 
Epidemics 

*  Nuclear  Radiation, 
Germ  Warfare, 

Toxic  Gases 

B 

A 

T 

T 

L 

E 

F 

1 

E 

L 

0 

SMOKE 

Vegetation  Fires 

Target  Hits 

DUST 

Dust  Storm 

Bomb  Blast 

DIRT,  SAND 

Sand  Storm 

Bomb  Blast 

OBSCURANTS 

Clouds,  Rain, 

Fog,  Snow, 

Haze,  Sand,  Dust 

*  Smoke  Canisters, 
Flares,  Battle 

Dust  and  Debris 

*  Enemy  actions  or  countermeasures  that  impact  on  survivability  or  susceptibility  are 
evaluated  as  components  of  operational  effectiveness  and  are  not  addressed  under 
operational  suitability. 
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PARAMETERS 

Environmental  pa-aroeters  can  be  used  to  characterize  the  intended  use  environment,  e.g.,  terrain 
that  the  system  is  intended  to  travel  over,  or  they  can  be  used  to  characterize  the  system’s 
capability  versus  the  environment,  e.g.,  minimum  visibility  level  at  which  the  seeker  will  be 
capable  of  operating.  The  first  category  is  used  to  communicate  the  user's  environmental 
requirements,  while  the  second  communicates  die  system's  capability  within  the  environment 
Examples  of  parameters  that  communicate  the  environment  are: 

Terrain  Grade:  The  incline  a  ground  vehicle  should  be  able  climb,  given  its 

power  and  traction 

Water  Depth:  The  water  level  through  which  a  ground  vehicle  should  be  able  to 

pass 

Sea  State:  Ocean  wave  conditions  under  which  a  vessel  should  be  able  to 

perform  certain  mission  functions. 

Examples  of  parameters  that  communicate  the  system’s  capability  within  an  environment  are: 

Range:  The  detection  range  of  a  seeker  under  certain  specified  obscurant 

conditions 

Speed:  Vehicle  speed  over  specified  terrain  conditions. 

KEY  POINTS 

Most  systems  have  environmental  limitations. 

These  limitations  are  defined  as  the  degree  of  conditions  (e.g.,  weather,  sea  stale)  under  which 
the  system  will  not  be  able  to  operate  effectively.  That  a  system  will  not  be  able  to  perform  in 
extreme  adverse  conditions  is  an  accepted  fact  for  most  systems,  but  the  threshold  for  non¬ 
effectiveness  must  be  known  to  the  user. 

Environmental  limitations  should  be  quantified  and  understood. 

Early  in  the  acquisition  process,  the  ranges  of  conditions  under  which  the  system  must  be 
effective  should  be  clearly  established  by  the  user  or  the  user's  representative.  The  frequency  of 
occurrence  of  adverse  conditions  (e.g.,  weather)  that  will  limit  system  performance  must  be 
understood  to  permit  acquisition  decisions  and  appropriate  planning  by  the  Service.  For  each 
system,  available  data  should  be  used  to  quantify  the  frequency  of  occurrence  of  any  limiting 
conditions.  The  specified  range  of  acceptable  environmental  conditions  and  identification  of 
significant  limiting  factors  should  provide  an  important  consideration  in  the  Service’s  planning 
for  the  OT&E. 

The  system  requirements  documents  should  discuss  the  required  operating  environment 

The  requirements  document  must  address  the  intended  operating  environment  for  the  system. 
Under  what  conditions  (weather,  terrain,  vegetation,  etc.)  will  the  system  be  employed? 
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Limitations  to  system  operation  and/or  maintenance  should  be  projected  prior  to  OT. 

If  there  are  any  expected  limitations  to  either  the  scope  of  operations  or  the  system's  capability, 
then  these  should  be  documented  by  the  acquisition  agency  prior  to  Milestone  IL  The  planning 
for  operational  testing  should  identify  any  environmental  OOIs.  The  OT&E  needs  to  address  the 
environmental  conditions  such  that  any  limitation  on  capability  due  to  environmental  conditions 
is  either  verified  or  further  understood  and  defined. 

Personnel  who  operate  and  maintain  the  system  are  affected  by  the  environment 

The  ability  of  operators  and/or  maintenance  personnel  to  function  under  certain  environmental 
conditions  also  needs  to  be  addressed.  Personnel  usually  are  not  stressed  to  their  endurance 
levels,  but  the  impact  of  the  weather,  protective  clothing,  and  reduced  visibility  may  be  factors 
that  impact  the  efficient  use  of  a  system.  If  the  system  design  requires  critical  personnel 
interaction,  then  consideration  of  these  environmental  areas  should  be  part  of  operational  test 
planning. 

Systems  with  optical  sensors  can  have  limited  performance  in  some  environments. 

Systems  that  have  electro-optical,  infrared,  or  millimeter  wave  seekers,  or  that  require  visual 
sighting  by  the  operators,  may  have  limitations  when  used  in  areas  with  high  levels  of 
obscurants,  e.g.,  smoke,  dust,  etc.  Any  system  limitations  that  arc  postulated  need  to  be 
estimated  and  included  in  program  planning  information.  The  level  of  the  limitations  should  be 
identified  as  a  Critical  Operational  Issue  if  the  limitation  is  critical  to  the  eventual  use  of  the 
system  in  its  intended  environment 

Environmental  conditions  at  the  OT  sites  usually  are  limited. 

The  sites  for  operational  testing  usually  are  limited  by  available  funding.  Normally  there  will 
only  be  one  site  for  the  early  operational  testing  and  this  site  is  more  likely  to  be  selected  for 
instrumentation,  test  facilities  or  ranges,  or  test  organizations  than  for  weather,  terrain,  or 
vegetation  conditions  that  are  representative  of  the  intended  operational  environment  System 
requirements  should  clearly  state  if  different  or  additional  environmental  conditions  are 
important  to  understanding  the  system's  operational  effectiveness  or  suitability.  For  example, 
terrain  testing  is  usually  pan  of  DT;  if  specific  problems  are  expected  with  terrain  or  vegetation, 
then  shon  operational  test  phases  or  demonstrations  should  be  considered  to  address  these  areas. 

Operational  testing  usually  is  not  performed  outside  the  system’s  intended  environmental 
operating  envelope. 

The  planning  of  the  operational  test  program  should  address  the  intended  operating  environment 
and  generally  should  not  incorporate  plans  to  operate  the  system  outside  that  environment 

Operational  testing  may  determine  additional  environmental  limitations. 

If  the  potential  for  additional  limitations  is  great  then  OT&E  also  must  attempt  to  define  any 
additional  environmental  limitations  on  the  system's  performance.  For  example,  a  system  may 
be  found  to  be  not  maintainable  under  chemical  attack.  If  the  item  must  be  decontaminated  prior 
to  some  maintenance  tasks,  and  the  capability  to  do  this  does  not  exist,  the  system  has  an  obvious 
major  limitation.  At  Milestone  in,  test  results  should  be  adequate  to  verify  the  level  of  any 
environmental  limitation. 
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2.4  ELECTROMAG  NETIC  ENVIRON  MENTAL  ft 
EFFECTS  (E3)  ^  :  . 

Electromagnetic  Environmental  Effects  (E3)  is  defined  as 

the  impact  of  the  electromagnetic  environment  upon  the  operational 
capability  of  military  forces,  equipment,  system,  and  platforms.  It 
encompasses  all  electromagnetic  disciplines,  including  electromagnetic 
compatibility/electromagnetic  Interference;  electromagnetic  vulner¬ 
ability;  electromagnetic  pulse;  electronic  counter-countermeasures; 
hazards  of  electromagnetic  radiation  to  personnel,  ordnance,  and 
volatile  materials;  and  natural  phenomena  effects  of  lightning  and 
p-statlc. 

Compatibility  with  the  electromagnetic  environment  is  an  important  issue  in  the  system’s  overall 
compatibility.  E3  includes  the  subjects  of  electromagnetic  interference  (EMI)  and 
electromagnetic  compatibility  (EMC).  Within  the  operational  suitability  area,  these  subjects  are 
examined  as  they  relate  to  companion  or  friendly  systems.  Vulnerabilities  to  enemy  electronic 
systems  are  addressed  under  operational  effectiveness.  To  properly  assess  the  E3  area  requires 
the  consideration  of  many  unusual  situations  that  may  cause  incompatibilities  within  the  E3 
areas.  Understanding  these  situations  requires  experience  and  knowledge  of  system  operation 
and  system  use  in  the  intended  operational  environment  Are  there  unforeseen  items  in  the 
environment  that  will  cause  problems  with  the  E3  conditions?  Tiat  companion  systems  need  to 
be  considered?  Are  there  unusual  situations  in  the  system’s  use  that  will  place  it  with  other 
systems  that  have  an  E3  consideration? 

E3  addresses  the  extent  to  which  a  system's  performance  is  degraded  by  electromagnetic  effects 
due  to  its  proximity  to  another  electronic  system.  EMC  and  EMI  are  evaluated  for  their  impact 
on  the  electromagnetic  transmissions  of  multiple  interfacing  systems,  as  well  as  for  their  impact 
on  friendly  systems  for  which  interfacing  is  not  intended.  Specifically,  if  two  systems  have 
electrical  transmissions  and  are  not  in’egrated,  but  are  brought  into  proximity  in  operational  use, 
then  consideration  of  their  mutual  operation  in  the  presence  of  each  other  is  a  part  of  EMC.  EMI 
addresses  interference  of  components  within  the  same  system. 

PARAMETERS 

While  discrete  engineering  parameters  (e.g ,  spurious  emission  levels,  radiation  leakage,  interfer- 
ence-to-noise  ratios)  are  quantitative  and  measurable  in  the  development  environment,  the  focus 
during  Operational  Testing  is  at  a  higher  total  system  and  environment  view.  Parameters  must 
be  focused  more  on  the  external  relationships,  as  well  as  the  internal  relationships.  For  example. 
Operational  Testing  might  verify  that  vastly  different  systems  and/or  multiple  copies  of  the  same 
system  can  operate  suitably  while  in  close  quarters  at  a  common  site.  The  objective  of  the  test 
would  be  to  determine  if  co-site  interference  problems  exist,  which  would  require  the  operator  to 
turn  one  system  off  when  using  another  system. 
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KEY  POINTS 

When  viewed  as  technical  considerations,  E3  problems  can  be  overcome. 

The  principal  risk  area  in  E3  is  associated  with  overlooking  some  potential  E3  condition,  and 
then  not  discovering  the  problem  until  late  in  the  development  process,  or  until  the  system  is 
fielded.  DT  will  examine  many  aspects  of  E3.  The  role  of  operational  testing  is  to  provide  a 
realistic  E3  environment  and  thereby  identify  any  potential  problems  that  were  not  identified  ear¬ 
lier  in  the  system's  development. 

Susceptibility  to  enemy  systems  usually  is  adequately  evaluated  under  operational 
effectiveness.  Compatibility  with  friendly  systems  is  not  adequately  addressed. 

Adequate  attention  generally  is  given  to  assuring  that  the  system  being  developed  is  not 
susceptible  to  interference  from  enemy  or  foreign  equipment  Attention  also  must  be  given  to 
compatibility  with  friendly  systems.  Examples  include 

•  other  systems  that  are  employed  by  the  user  organization  or  the  same  military 
Service  and  arc  placed  in  proximity  to  the  system  being  developed.  For  exam¬ 
ple,  other  Army  systems  used  near  the  Army  system  under  development. 

•  other  systems  that  may  be  used  in  proximity  to  the  system  being  developed  by 
other  militaiy  Services.  For  example,  are  there  USMC  or  Air  Force  ground 
electronics  systems  that  will  be  used  in  proximity  to  an  Army  ground  electronics 
system  that  is  being  developed? 

•  other  systems  under  development  Compatibility  may  be  examined  during 
OT&E  with  the  systems  that  are  existing  in  the  operating  units,  but  are  there 
other  systems  in  development  that  will  be  major  factors  in  the  E3  environment  of 
the  new  system.  Are  these  items  included  in  the  planned  operational  testing? 

The  operational  testing  environment  needs  to  represent  the  total  E3  environment  to  the  maximum 
extent  possible. 

Complementary  systems  and  unusual  conditions  need  to  be  included  in  the  E3  assessment 

Situanons  should  be  identified  where  systems  are  used  to  compliment  each  other  in  ways  that  are 
not  considered  the  norm,  but  which  are  pan  of  the  expected  system  capability.  Joint  operations  of 
systems  by  two  or  more  of  the  military  Services  are  likely  to  introduce  situations  that  need  to  be 
examined.  Other  E3  conditions  may  result  from  operations  in  unusual  environmental  conditions 
(e.g.,  weather  terrain).  Surface  ships  operating  in  high  sea  states  may  have  E3  environments  that 
are  different  h  l  anticipated  because  of  the  ships  attitude  at  various  times.  These  conditions 
need  to  be  exar.-r.ed  and  considered  during  the  planning  for  OT&E. 

Friendly  compatible  systems  must  be  identified. 

The  criticality  of  these  systems  must  be  known  so  that  limited  test  resources  can  be  focused  on 
examining  the  E3  environment  with  these  items.  Does  the  test  plan  list  the  friendly  systems  to  be 
included  in  the  test,  as  well  as  the  systems  that  are  not?  At  Milestone  IIIA,  there  will  be  an 
initial  assessment  of  the  E3  risk  of  the  system.  An  early  assessment  may  be  possible  by 
examining  available  E3  area  technical  test  data.  At  Milestone  III,  the  compatibility  with  the 
friendly  systems  should  be  known  and  the  risks  areas  identified.  The  operational  testing  should 
have  included  all  of  the  systems  that  were  planned  to  be  part  of  the  testing. 
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2.$  SOFTWARE  $L 
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For  the  purposes  of  this  guide,  software  supportability  is  defined  as 

a  measure  of  the  adequacy  of  products,  resources,  and  procedures  that 
are  needed  to  support  the  software  component  of  a  system. 


Software  support  activities  are  necessary  to  establish  an  operational  baseline,  install  the  software 
in  the  system,  modify  or  update  the  software,  and  meet  the  users'  requirements.  Software 
supportability  is  a  function  of  the  quality  of  the  software  products  themselves,  the  capabilities  of 
the  software  support  resources,  and  the  adequacy  of  the  life  cycle  processes  that  affect  the 
procurement,  development,  modification,  and  operational  support  of  the  software. 

The  criticality  of  the  software  supportability  is  best  exemplified  by  the  need  for  some  system 
software  to  be  periodically  revised  or  updated  to  correspond  to  new  situations.  Electronic 
warfare  systems  are  periodically  updated  as  new  information  on  threats  is  received  or  new  tactics 
are  implemented.  The  ability  to  revise  the  software  in  a  timely  and  efficient  manner  can  be 
critical  to  the  suitability  of  the  system  to  perform  its  required  mission. 

PARAMETERS 

The  methods  for  assessing  die  suitability  of  system  software  have  evolved  over  the  last  ten  years. 
Most  of  the  activity  by  the  OTAs  in  this  area  has  resulted  in  qualitative  evaluation  methods  using 
questionnaires,  with  the  results  being  converted  by  scoring  methods  into  quantitative  measures. 
For  example,  maintainability  evaluation  of  the  software  for  a  specific  system  might  be  scored  as 
a  "C"  This  means  that  the  average  qualitative  evaluation  of  the  software  against  a 
maintainability  evaluation  questionnaire  resulted  in  a  judgment  that  the  software  "generally 
agreed”  with  the  statements  in  the  maintainability  questionnaire.  A  range  of  responses  is 
possible:  “A”  =  completely  agree,  “B”  =  strongly  agree,  “C”  =  generally  agree,  “D”  =  generally 
disagree,  “E”  =  strongly  disagree,  and  “F”  -  completely  disagree. 

In  some  software  evaluations,  maturity  levels  also  arc  reported.  The  maturity  levels  may  be 
expressed  in  the  "number  of  software  changes"  or  in  "software  change  points."  The  term 
“software  change  points”  is  used  when  the  individual  software  changes  that  are  identified  as 
being  needed  arc  weighted  by  the  severity  of  their  operational  impact  on  the  system.  This 
multiplication  results  in  a  parameter  termed  "software  change  points.” 

KEY  POINTS 

The  software  documentation  can  be  the  key  to  the  effective  support  of  the  software. 

The  timely  delivery  of  software  documentation  is  a  key  element  in  allowing  the  life  cycle  sup¬ 
port  activity  to  maintain  and  upgrade  the  software.  Review  of  the  documentation  has  been  done 
in  the  past  using  a  checklist  approach.  This  approach  was  useful  when  the  software  was  of  a 
limited  scope.  However,  with  more  complex  systems,  the  amount  of  software  is  such  that  a 
manual,  exhaustive  inspection  is  no  longer  possible.  Sampling  techniques  or  other  approaches 
that  are  to  be  used  need  to  be  identified  in  the  TEMP  and  the  test  plan. 
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The  maintainability  of  software  depends  on  its  design  and  arrangement 

The  characteristics  of  the  software  that  determine  its  maintainability  include  its  modularity, 
descriptiveness  of  the  software  code  and  documentation,  consistency  throughout  the  code  and 
documentation,  simplicity,  expandability  (is  the  code  built  with  the  objective  of  making  it  easy  to 
expand?),  and  instrumentation  (does  the  code  allow  the  easy  use  of  testing  aids?).  These 
characteristics  have  been  evaluated  by  the  OTAs  primarily  through  the  use  of  review 
questionnaires.  The  application  of  the  questionnaires  to  large  software  packages,  the  selection  of 
samples  to  examine,  and  the  relationship  of  the  sample  results  to  assessment  of  the  entire  system 
software  are  risk  areas  that  need  to  be  examined  during  the  test  planning  and  execution. 

The  interface  that  the  software  presents  can  be  critical  to  system  operation. 

The  displays,  menus,  etc.,  that  the  system  software  creates  are  at  the  critical  junction  between  the 
computer-driven  system  and  its  user  or  maintainer.  The  principal  evaluation  methods  relate  to 
the  reaction  of  the  person  to  the  displays.  Does  the  software  present  clear  and  understandable  in¬ 
formation?  Is  there  a  consistency  throughout  the  systems  operation,  e.g.,  similar  menu  usage, 
key  stroking  similar  in  similar  situations,  complex  tasks  have  easily  understood  sequence,  etc.?’ 
The  evaluation  of  the  user-software  interface  may  be  by  questionnaire  oi  by  qualitative 
assessment  at  debriefings.  The  questionnaire  method  results  in  more  consistent  and  perhaps 
more  thorough  evaluations,  but  the  unstructured  reaction  of  the  personnel  involved  should  not  be 
ignored.  Dissimilarity  with  the  predecessor  system  may  cause  negative  reactions. 

The  ability  to  maintain  and  modify  the  software  depends  on  the  adequacy  of  the  software 
support  resources. 

Evaluation  of  the  planned  software  support  resources  consists  of  the  evaluation  of  the  support 
personnel,  support  systems,  and  facilities  that  are  planned.  Evaluation  of  personnel  consists 
primarily  of  identifying  the  number  of  people  and  the  skill  levels  needed  to  provide  the  required 
support.  The  software  support  system  comprises  the  computers  and,  in  some  cases,  unique 
software  that  is  needed  to  provide  software  maintenance,  modification,  and  upgrades. 

The  maturity  of  the  software  can  be  evaluated  by  examining  the  faults  or  errors  that  have 
been  found  and  the  status  of  the  individual  corrective  actions. 

As  the  software  is  tested,  errors  or  faults  are  found.  Some  OTAs  evaluate  each  software  fault 
(i.e„  weight  each  fault  by  the  severity  of  the  effect  on  the  system)  and  produce  a  plot  of  the 
cumulative  number  changes,  or  change  points,  against  the  amount  of  testing  that  has  been 
performed.  This  curve  generally  shows  an  increasing  number  of  faults  as  the  software  is 
exercised,  followed  by  a  decreasing  rate  indicating  that  most  of  the  faults  have  been  found.  The 
use  of  seventy  weighting  assures  that  inordinate  weight  is  not  given  to  a  number  of  minor  faults, 
while  major  faults  are  ignored.  The  plot  of  software  maturity  can  give  a  clear  view  of  when  the 
software  is  reaching  a  maturity,  when  it  is  reasonable  to  start  OT,  and  when  the  software  is 
mature  enough  to  enter  the  operational  inventory.  The  DOT&E  staff  assistant  should  be  familiar 
with  this  software  maturity  technique  if  it  is  to  be  used  on  one  of  the  assigned  programs. 

Software  maturity  depends  on  testing  exposure. 

The  assessment  of  the  maturity  of  the  software  depends  on  the  thoroughness  of  the  software 
testing.  Are  all  software  capabilities  being  tested?  Arc  the  low  probability  paths,  as  well  as  the 
nominal  conditions,  being  exercised?  Are  error  routines  and  fault  identification  modules  being 
included  in  the  testing?  To  accurately  judge  the  maturity  of  the  software,  an  attempt  should  be 
made  to  assure  that  all  aspects  of  the  software  are  included  in  the  test 
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Chapter  3 

SERVICE  OTAs  COMMON  RELIABILITY,  AVAILABILITY, 
AND  MAINTAINABILITY  (RAM)  TERMINOLOGY 


The  Military  Services  Operational  Test  and  Evaluation  Agencies  (OTAs)  have  agreed  to 
common  methods  to  be  used  during  testing  that  involve  more  that  one  Service  This  is  defined  as 
Muln-Service  OT&E.  In  1989,  the  four  Service  OTAs  agreed  to  a  listing  of  "Common 
Reliability,  Availability  and  Maintainability  Terms  for  use  in  Multi-Service  OT&E  Test 
Programs."  This  listing  is  contained  in  Annex  A  to  the  OTAs  Memorandum  of  Agreement  and  is 
included  in  the  Operational  Suitability  Guide  for  reference. 
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ANNEX  A 


COMMON  RELIABILITY,  AVAILABILITY,  AND 
MAINTAINABILITY  (RAM)  TERMINOLOGY 

1.  Purpose.  This  Annex  provides  the  policy  and  common  RAM 
terminology  for  the  quantitative  portion  of  MOT6E  suitability 
evaluations . 

2.  Background .  MOTSE  common  terms  are  intended  to  convey  the 
same  meaning  to  all  Services.  Therefore,  they  avoid  terms  used 
elsewhere  with  different  meanings.  Existing  terms  used  by  one 
or  more  Services  were  selected  when  possible.  Table  A-l  com¬ 
pares  the  RAM  terms  to  be  used  for  multiservice  testing,  as 
described  in  this  Annex,  with  the  relative  service-unique  RAM 
terms  currently  in  use.  Other  relevant,  service-unique  RAM 
terms  are  provided  in  appendices  to  this  Annex. 

3 .  Policy 

a.  Common  terms  described  in  this  Annex  will  be  used  as 
appropriate  in  multiservice  test  reports.  If  additional  terms 
are  necessary,  they  should  be  agreed  upon  and  clearly  defined 
by  all  participating  agencies. 

b.  Multiservice  terms  selected  will  be  included  in  the 
multiservice  TEMP. 

4 .  Common  RAM  Terms  for  MOT&E 


a.  Reliability.  Reliability  consists  of  two  major  areas? 
mission  reliability  and  logistics  support  frequency. 

(X)  Mission  reliability  is  the  probability  a  system  can 
complete  its  required  operational  mission  without  an  operational 
mission  failure  (OMF) .  An  OMF  is  a  failure  that  precludes 
successful  completion  of  a  mission,  and  must  be  specifically 
defined  for  each  system.  For  some  systems,  mission  reliability 
may  be  better  expressed  as  Mean  Time  (miles,  rounds,  etc.) 
Between  Operational  Mission  Failure  (MTBOMF) .  (See  paragraph  5 
for  definition. ) 

(2)  Logistics  support  frequency  is  a  representative 
time  between  incidents  requiring  unscheduled  maintenance, 
unscheduled  removals,  and  unscheduled  demands  for  spare  parts, 
whether  or  not  mission  capability  is  affected.  Logistics  sup¬ 
port  frequency  may  be  expressed  as  Mean  Time  Between  Unscheduled 
Maintenance  (MTBUM) .  (See  paragraph  5  for  definition.) 
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b.  Maintainability.  Maintainability  consists  of  three 
major  areas:  OMF  repair  time,  corrective  maintenance  time,  and 
maintenance  ratio.  Maintainability  may  be  expressed  as  (1)  Mean 
Operational  Mission  Failure  Repair  Time  (MOMFRT) ,  (2)  Mean 
Corrective  Maintenance  Time  (MCMT) ,  (3)  Maximum  Time  To  Repair 
(MaxTTR) ,  and  (4)  various  maintenance  ratios,  e.g..  Maintenance 
Man-Hours  Per  Operating  Hour,  Mile,  Round,  etc.  (See  paragraph 
5  for  definitions.) 

c.  Availability.  Operational  availability  (Aq)  is  the 

probability  that  a  system  will  be  ready  for  operational  use 
when  required.  (See  paragraph  5  for  definition.) 

d.  Diagnostics.  Diagnostics  addresses  the  ability  of 
integrated  diagnostics  (automated,  semi-automated,  and  manual 
techniques  taken  as  a  whole)  to  fault-detect  and  fault-isolate 
in  a  timely  manner.  Diagnostics  may  be  expressed  as  (1)  the 
percent  of  correct  detection  given  that  a  fault  has  occurred 
(Pcd),  and  (2)  Mean  Time  To  Fault  Locate  (MTTFL) .  (See  para¬ 
graph  5  for  definitions.) 

5.  Definitions  for  Multiservice  Terms 


a.  Mean  Time  Between  Operational  Mission  Failures 
(MTBOMF)  :•  The  total  operating  time  (e.g.,  driving  time,  flying 
time,  or  system-on  time)  divided  by  the  total  number  of  OMFs. 

b.  Mean  Time  Between  Unscheduled  Maintenance  (MTBUM) :  The 
total  operating  time  divided  by  the  total  number  of  incidents 
requiring  unscheduled  maintenance. 

c.  Mean  Operational  Mission  Failure  Repair  Time  (MOMFRT) : 
The  total  number  of  clock-hours  of  corrective,  on-system, 
active  repair  time,  which  is  used  to  restore  failed  systems  to 
mission-capable  status  after  an  operational  mission  failure 
(OMF)  occurs,  divided  by  the  total  number  of  OMFs. 

d.  Mean  Corrective  Maintenance  Time  (MCMT) :  The  total 
number  of  clock-hours  of  corrective,  on-system,  active  repair 
time  due  to  all  corrective  maintenance  divided  by  the  total 
number  of  incidents  requiring  corrective  maintenance. 

e.  Maximum  Time  To  Repair  (MaxTTR)  :■  That  time  below  which 
a  specified  percentage  of  all  corrective  maintenance  tasks  must 
be  completed. 

f.  Maintenance  Man-Hours  Per  Operating  Hour  (MMH/OH):  The 
cumulative  number  of  maintenance  man-hours  during  a  given 
period  divided  by  the  cumulative  number  of  operating  hours.  If 
appropriate,  other  terms  such  as  miles  or  rounds  may  be  sub- 
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stituted  for  hours.  Scheduled  as  well  as  corrective  main¬ 
tenance,  in  keeping  with  the  users  maintenance  requirements, 
are  included  without  regard  to  their  effect  on  mission  or 
availability  of  the  system. 

g.  Operational  Availability  (Ao):  A0  is  either  the  total 
uptime  divided  by  the  uptime  plus  downtime  when  operated  in  an 
operational  mission  scenario,  or  the  number  of  systems  that  are 
ready  divided  by  the  number  possessed. 

h.  Percent  of  Correct  Detection  Given  That  a  Fault  Exists 
(Pcd):  The  number  of  correct  detections  divided  by  the  total 
number  of  confirmed  faults. 

i.  Mean  Time  To  Fault  Locate  (MTTFL)  :■  Tne  total  amount  of 
time  required  to  locate  faults  divided  by  the  total  number  of 
faults. 


APPENDICES : 

1  -  Army  Terms  and  Definitions 

2  -  Navy  Terms  and  Definitions 

3  -  Marine  Corps  Terms  and  Definitions 

4  -  Air  Force  Terms  and  Definitions 
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TABLE  A— 1 .  COMPARISON  OF  MULTISERVICE  AND  SERVICE-UNIQUE  TERMS 


MULTI- 


CATEGORY 

SERVICE 

ARMY 

NAVY 

AIR  FORCE 

MARINES 

RELIABILITY 

MTBOMF 

MTBOMF 

MTBMCF 

MTBCF 

MTBOMF 

MTBUM 

MTBUMA 

MTBF 

MTBME 

MTBUMA 

MAINTAINABILITY 

MOMFRT 

NONE 

MTTR 

NONE 

NONE 

MCMT 

MTTR 

NONE 

MRT 

MTTR 

MaxTTR 

MaxTTR 

NONE 

MaxTTR 

MaxTTR 

MMH/OH 

MR 

DMMH/FH 

MMH/OH 

MR 

AVAILABILITY 

Ao 

Ao 

Ao 

Ao 

AND 

OTHERS 

Ao 

DIAGNOSTICS 

Pcd 

Pcd 

Pcd 

Pcd 

Pcd 

MTTFL 

AND 

OTHERS 

AND 

OTHERS 

AND 

OTHERS 

AND 

OTHERS 
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ARMY  TERMS  AND  DEFINITIONS 

1.  Purpose.  This  Appendix  provides  the  RAM  terms  and 
definitions  most  relevant  to  this  Ann  :t  and  used  within  the 
Army  in  conducting  and  reporting  OT&E  activity  in  accordance 
with  AR  702-3.  They  are  included  in  the  Memorandum  of 
Agreement  so  as  to  assist  other  Services  in  understanding  RAM 
terms  as  used  by  the  Army.  Terms  used  by  other  Services  are 
included  in  Appendices  2,  3,  and  4. 

2.  Definitions 


a.  Durability:  A  special  case  of  reliability;  the 
probability  that  an  item  will  successfully  survive  to  its 
projected  life,  overhaul  point,  or  rebuild  point  (whichever  is 
the  more  appropriate  durability  measure  for  the  item)  without  a 
durability  failure.  (See  Durability  Failure.) 

b.  Failure:  The  event,  or  inoperable  state,  in  which  an 
item  or  part  of  an  item  does  not,  or  would  not,  perform  as 
previously  specified.  (See  MIL-STD  721.) 

c.  Failure,  Critical:  A  failure  (or  combination  of 
failures)  that  prevents  an  item  from  performing  a  specified 
mission.  (Note:  Normally  only  one  failure  may  be  charged 
against  one  mission.  Critical  failure  is  related  to  evaluation 
of  mission  success.) 

d.  Failure,  Durability:  A  malfunction  that  precludes 
further  operation  of  the  item,  and  is  great  enough  in  cost, 
safety,  or  time  to  restore,  that  the  item  must  be  replaced  or 
rebuilt. 


e.  Failure  Mode:  The  mechanism  through  which  failure 
occurs  in  a  specified  component  (for  example,  short,  open, 
fatigue,  fracture,  or  excessive  wear).  (See  MIL-STD  721.) 

f-  Inherent  RAM  Value:-  Any  measure  of  RAM  that  includes 
only  the  effects  of  an  item  design  and  its  application,  and 
assumes  an  ideal  operating  and  support  environment. 

g.  Maintainability:  A  measure  of  the  ability  of  an  item 
to  be  retained  in,  or  restored  to,  a  specified  condition  when 
maintenance  is  performed  by  personnel  having  specified  skill 
levels  using  prescribed  procedures  and  resources . 

h.  Maintenance  Ratio  (MR) :  A  measure  of  the  maintenance 
manpower  required  to  maintain  a  system  in  an  operational  envi¬ 
ronment.  It  is  expressed  as  the  cumulative  number  of  direct 


A-l-l 


Chapter  3 


Page  3-7 


maintenance  man-hours  (see  AR  570-2)  during  a  given  period, 
divided  by  the  cumulative  number  of  system  life  units  (such  as 
hours,  rounds,  or  miles)  during  the  same  period.  The  MR  is 
expressed  for  each  level  of  maintenance  and  summarized  for 
combined  levels  and  maintenance.  All  maintenance  actions  are 
considered  (that  is,  scheduled  as  well  as  corrective,  and 
without  regard  to  their  effect  on  mission  or  availability  of 
system).  Man-hours  for  off-system  repair  of  replaced 
components  are  included  in  the  MR  for  the  respective  level . 

i.  Maximum  Time  To  Repair  (MaxTTR)  :■  That  time  below 
which  a  specified  percentage  of  all  corrective  maintenance 
tasks  must  be  completed.  When  stated  as  a  requirement,  the 
MaxTTR  should  be  stated  for  organizational  and  direct  support 
levels  of  maintenance.  MaxTTR  is  used  as  an  "on-system" 
maintainability  parameter;  it  is  not  used  for  the  off-system 
repair  of  replaced  components . 

j .  Mean  Time  Between  Essential  Maintenance  Actions 
(MTBEMA) :  For  a  particular  measurement  interval,  the  total 
number  of  system  life  units  divided  by  the  total  number  of 
nondeferrable  maintenance  actions.  This  parameter  indicates 
the  frequency  of  demand  for  essential  maintenance  support  and 
includes  incidents  caused  by  accidents,  maintenance  errors,  and 
item  abuse.  (Not  included  are  crew  maintenance  completed 
within  a  specified  number  of  minutes,  maintenance  deferrable  to 
the  next  scheduled  maintenance,  system  modification,  and  test- 
peculiar  maintenance.) 


k.  Mean  Time  Between  Operational  Mission  Failure  (MTBOMF) : 
A  measure  of  operational  effectiveness  that  considers  the 
inability  to  perform  one  or  more  mission-essential  functions. 


1.  Mean  Time  Between  Unscheduled  Maintenance  Actions:' 
Computed  by  the  following  formula:- 


MTBUMA  = 


Operating  time 


Total  number  of  unscheduled  maintenance  actions 


m.  Mean  Time  To  Repair  (MTTR) :  The  sum  of  corrective 
maintenance  times  divided  by  the  total  number  of  corrective 
maintenance  actions  during  a  given  period  of  time  under  stated 
conditions.  MTTR  may  be  used  to  quantify  the  systems  maintain¬ 
ability  characteristic.  MTTR  applies  to  the  system-level 
configuration;  it  will  be  used  as  an  "on-system"  maintainability 
index  and  not  for  the  repair  of  components.  MTTRs  will  be 
stated  for  the  unit  and  the  intermediate  direct  support  levels 
of  maintenance  along  with  the  percentage  of  actions  repaired  at 
each  level . 
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n.  Mission  Reliability  (Rm):  A  measure  of  operational 
effectiveness.  It  is  stated  in  terms  of  a  probability  of 
completing  a  specified  mission  profile  or  the  mean  time  (or 
distance  or  rounds)  between  critical  failures. 

o.  Mission-Essential  Functions:  The  minimum  operational 
tasks  that  the  system  must  be  capable  of  performing  to 
accomplish  its  mission  profiles. 

p.  Off-System  Maintenance:  Maintenance  associated  with 
the  diagnosis  and  repair  of  components  for  return  to  stock. 

q.  On-System  Maintenance  :•  Maintenance  necessary  to  keep  a 
system  in,  or  return  a  system  to,  an  operating  status. 

r.  Operational  Availability:  The  proportion  of  time  a 
system  is  either  operating,  or  is  capable  of  operating,  when 
used  in  a  specific  manner  in  a  typical  maintenance  and  supply 
environment.  All  calendar  time  when  operating  in  accordance 
with  wartime  operational  mode  summary/mission  profile  (OMS/MP) 
is  considered.  The  formula  is  as  follows: 

OT  +  ST 

flo  = 

OT  +  ST  +  TCM  +  TPM  +  TALDT 

Total  calendar  time  minus  total  downtime 


Total  calendar  time 

Where:  OT  =  The  operating  time  during  OMS/MP 

ST  =  Standby  time  (not  operating,  but  assumed  operable) 
during  OMS/MP 

TCM  =  The  total  corrective  maintenance  downtime  in 
clock-hours  during  OMS/MP 

TPM  =  The  total  preventive  maintenance  downtime  in 
clock-hours  during  OMS/MP 

TALDT  =  Total  administrative  and  logistics  downtime 
(caused  by  OMFs)  spent  waiting  for  parts, 
maintenance  personnel,  or  transportation  during 
OMS/MP 

Other  forms  of  this  equation  are  substituted  depending  on  the 
system  type  (see  AMC/TRADOC  PAM  70-11)  such  as  the  inclusion  of 
relocation  time. 

s.  Operational  Mission  Failure  (OMF):  Any  incident  or 
malfunction  of  the  system  that  causes  (or  could  cause)  the 
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inability  to  perform  one  or  more  designated  mission-essential 
functions . 

t.  Operational  RAM  Value:-  Any  measure  of  RAM  that 
includes  the  combined  effects  of  item  design,  quality, 
installation,  environment,  operation,  maintenance,  and  repair. 
(This  measure  encompasses  hardware,  software,  crew,  maintenance 
perso.aiel,  equipment  publications,  tools,  TMDE,  and  the 
natural,  operating,  and  support  environments. 

u.  Reliability:'  The  probability  that  an  item  can  perform 
its  intended  functions  for  a  specified  time  interval  under 
stated  conditions. 

v.  Reliability  After  Storage:-  This  may  be  a  stated 
requirement.  If  appropriate,  it  specifies  the  amount  of 
deterioration  acceptable  during  storage.  Length  of  storage, 
storage  environment,  and  surveillance  constraints  are 
identified.  This  requirement  may  not  be  testable;  it  may  rely 
on  an  engineering  analysis  for  its  assessment  before  deployment. 

w.  System  Readiness  Objective  (SRO):-  One  of  a  group  of 
measures  relating  to  the  effectiveness  of  an  operational  unit 
to  meet  peacetime  deployability  and  wartime  mission  require¬ 
ments  considering  the  unit  set  of  equipages  and  the  potential 
support  assets  and  resources  available  to  influence  the  units 
operational  readiness  and  sustainability.  Peacetime  and 
wartime  SROs  will  differ  due  to  usage  rate,  OMS/MPs,  and 
operational  environments.  Examples  of  SROs  include; 
operational  availability  at  peacetime  usage  rates,  operational 
availability  at  wartime  usage  rates,  sortie  generations  per 
given  time  frame  (aircraft),  and  maximum  ALDT  (intermittent 
mission) .  SROs  must  relate  quantitatively  to  system  design 
parameters  (RAM)  and  relate  to  support  resource  requirements. 
(See  AR  700-127. ) 
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NAVY  TERMS  AND  DEFINITIONS 


1.  Purpose.  This  Appendix  provides  the  RAM  terms  and 
definitions  most  relevant  to  this  Annex  and  used  within  the 
Navy  in  conducting  and  reporting  OT&E  activity  in  accordance 
with  COMOPTEVFOP. INST  3960.  IE.  They  are  included  in  the 
Memorandum  of  Agreement  so  as  to  assist  other  services  in 
understanding  RAM  terms  as  used  by  the  Navy.  Terms  used  by 
other  services  are  included  in  Appendices  1,  3,  and  4. 

2.  Definitions 


a.  Availability:  A  measure  of  the  degree  to  which  an  item 
is  in  an  operable  and  committable  state  at  the  start  of  a 
mission  when  the  mission  is  called  for  at  an  unknown  (random) 
time.  In  OT&E,  operational  availability  (A0)  is  the  usual 
measure.  (See  Operational  Availability.) 

b.  Failure.  Critical:  One  that  prevents  the  system  from 
performing  its  mission  or  results  in  the  loss  of  some 
significant  mission  capability. 

c.  Failure.  Minor:-  One  that  affects  system  performance 
but  does  not  impact  the  ability  to  perform  the  mission. 

d.  Maintainability:  The  capability  of  an  item  to  be 
retained  in  or  restored  to  specified  conditions  when 
maintenance  is  performed  by  personnel  having  specified  skill 
levels,  using  prescribed  procedures  and  resources,  at  each 
prescribed  level  of  maintenance  and  repair.  MTFL,  MTTR,  and 
MSI  are  frequently  calculated  in  maintainability  evaluations. 

e.  Maintenance  Support  Index  (MSI):  The  ratio  of  total 
man-hours  required  for  maintenance  (preventive  plus  corrective) 
to  the  total  operating  (up)  time.  Frequently  computed  as  part 
of  Test  S-2  Maintainabil:  ty. 

f .  Mean  Flight  Hours  Between  Failure/Mean  Time  Between 
Failure  (MFHBF/MTBF) :  See  Reliability. 

g.  Mean  Time  To  Fault-Locate  (MTFL):  The  total  fault- 
location  time  divided  by  the  number  of  critical/major  failures. 
Frequently  computed  as  part  of  Test  S-2  Maintainability. 
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h.  Mean  Time  To  Repair  (MTTR) :  Normally  computed  as  part 
of  maintainability,  MTTR  is  the  average  time  required  to 
perform  active  corrective  maintenance.  Corrective  maintenance 
is  the  time  during  which  one  or  more  personnel  are  repairing  a 
critical  or  major  failure  and  includes:  preparation,  fault 
location,  part  procurement  from  local  (on-board)  sources,  fault 
correction,  adjustment/calibration,  and  follow-up  checkout 
times.  It  excludes  off-board  logistic  delay  time. 

i.  Mission  Reliability:  See  Reliability. 

j.  Operational  Availability:  (See  Availability  for  basic 
definition.)  Operational  availability  is  computed  and  reported 
as  follows: 

(1)  Continuous-Use  Systems:  Operational  availability 
shall  be  designated  Aq  and  shall  be  determined  as  the  ratio  of 

system  "uptime''  to  system  "uptime  plus  downtime." 

(2)  "On-Demand"  Systems :  Operational  availability 
shall  be  designated  Aq(j  and  shall  be  determined  as  the  ratio 

of  the  "number  of  times  the  system  was  available  to  perform  as 
required  to  the  total  number  of  times  its  performance  was 
required."  (Note:  "Total  number  of  times  its  performance  was 
required"  shall  be  the  number  of  times  attempted  and  the  number 
of  times  it  was  operationally  demanded  but  not  attempted 
because  the  system  was  known  to  be  inoperable.) 

(3)  Impulse  Systems:  Operational  availability  shall 
be  designated  Aq^,  and  since  Aq^  and  R  are  inseparable 

during  testing,  only  reliability  (R)  shall  be  reported. 

k.  Operational  Effectiveness:  The  capability  of  a  system 
to  perform  its  intended  function  effectively  over  the  expected 
range  of  operational  circumstances,  in  the  expected  environment, 
and  in  the  face  of  the  expected  threat,  including  counter¬ 
measures  where  appropriate. 

l.  Operational  Suitability:  The  capability  of  the  system, 
when  operated  and  maintained  by  typical  fleet  personnel  in  the 
expected  numbers  and  of  the  expected  experience  level,  to  be 
reliable,  maintainable,  operationally  available,  logistically 
supportable  when  deployed,  compatible,  interoperable,  and  safe. 

m.  Reliability:'  The  duration  or  probability  of  failure- 
free  performance  under  stated  conditions.  In  OTSE,  reliability 
is  usually  reported  in  one  of  two  ways: 

(1)  Mean  Time  Between  Failure  (MTBF):  For  raore-or-less 
continuously  operated  equipment,  the  ratio  of  total  operating 
time  to  the  sum  of  critical  and  major  failures.  MTBF  is 
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sometimes  modified  to  mean  flight  hours  between  failures 
(MFHBF) . 


(2)  Mission  Reliability:  For  equipment  operated  only 
during  a  relatively  short-duration  mission  (as  opposed  to 
equipment  operated  more-or-less  continuously),  the  probability 
of  completing  the  mission  without  critical  or  major  failure. 
Frequently  expressed  as  exp  (-t/MTBF),  where  "t“  is  mission 
duration  and  MTBF  is  as  defined  above. 
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MARINE  CORPS  TERMS  AND  DEFINITIONS 


1.  Purpose.  This  Appendix  provides  the  RAM  terms  and 
definitions  most  relevant  to  this  Annex  and  used  within  the 
Marine  Corps  in  conducting  and  reporting  OT&E  activity  in 
accordance  with  FMFM  4-1  {Combat  Service  Support),  TRADOC/AMC 
Pamphlet  70-11  (RAM  Rationale  Report  Handbook),  and  DoD 
3235. 1-H  (Test  and  Evaluation  of  System  Reliability, 
Availability,  and  Maintainability,  a  Primer).  They  are 
included  in  the  Memorandum  of  Agreement  so  as  to  assist  other 
services  in  understanding  RAM  terms  as  used  by  the  Marine 
Corps.  Terms  used  by  other  services  are  included  in  Appendices 

1 ,  2 ,  and  4 . 

2.  Definitions 


a.  Achieved  Availability  (Aa):-  Computed  with  the 
following  formula: 

OT 

Aa  - - 

OT  +  TCM  +  TPM 

Where:-  OT  =  Operating  time 

TCM  =  Total  corrective  maintenance 
TPM  =  Total  preventive  maintenance 

b .  Administrative  and  Logistics  Downtime  (ALDT) :  The 
period  of  time  that  includes  (but  is  not  limited  to)  time 
waiting  for  parts,  processing  records,  and  transporting 
equipment  and/or  maintenance  personnel  between  the  using  unit 
and  repair  facility. 

c.  Corrective  Maintenance  (CM):  Maintenance  that  is 
performed  on  a  nonscheduled  basis  to  restore  equipment  to 
satisfactory  condition  by  correcting  a  malfunction  (unscheduled 
maintenance) .  The  measure  is  Total  Corrective  Maintenance 
(TCM)  time. 

d.  Depot  Level  Maintenance:  Maintenance  that  is  performed 
by  designated  industrial-type  activities  using  production-line 
techniques,  programs,  and  schedules.  The  principal  function  is 
to  overhaul  or  completely  rebuild  parts,  subassemblies, 
assemblies,  or  the  entire  end  item. 

e.  Essential  Maintenance  Action  (EMA) :  Maintenance  that 
must  be  performed  prior  to  the  next  mission.  This  includes 
correcting  operational  mission  failures,  as  well  as  performing 
certain  unscheduled  maintenance  actions. 
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f.  Failure:  Any  single,  combination,  or  summation  of 
hardware  or  software  malfunctions  that  cause  a  maintenance 
action  to  be  performed. 

g.  Inherent  Availability  (Ai):  Computed  with  the 
following  formula: 

OT 

Ai  - - 

OT  +  TCM 

h.  Intermediate  Level  Maintenance  (ILM):  Maintenance  that 
is  authorized  by  designated  maintenance  activities  in  support 
of  using  organizations.  The  principal  function  of  ILM  is  to 
repair  subassemblies,  assemblies,  and  major  items  of  equipment 
for  return  to  a  lower  echelon  or  to  supply  channels.  Measures 
are  as  follows: 

(1)  MTTR  at  ILM 

(2)  MaxTTR  at  ILM 

i.  Maintenance  Ratio  (MR) :  Computed  by  the  following 
formula: 


Total  man-hours  of  maintenance 

MR  - - 

Operating  time 

j.  Maximum  Time  To  Repair  (MaxTTR) :  That  time  below  which 
a  specified  percentage  of  all  corrective  maintenance  tasks  must 
be  completed. 

k.  Mean  Time  Between  Operational  Mission  Failure  (MTBOMF) 
Computed  by  the  following  formula: 

Operating  time 

MTBOMF  =  - 

Total  number  of  operational  mission  failures 

l.  Mean  Time  Between  Unscheduled  Maintenance  Actions 
(MTBUMA) :  Computed  by  the  following  formula: 


MTBUMA 


Operating  time 

Total  number  of  unscheduled  maintenance  actions 


m.  Mean  Time  To  Repair  (MTTR) :  Computed  by  the  following 
formula: 


MTTR  = 


Total  corrective  maintenance  time 


Total  number  of  corrective  maintenance  actions 
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n.  Operational  Availability  (Ao)  :•  Computed  by  the 
following  formula: 


OT  +  ST 

Ao  = 

OT  +  ST  +  TCM  +  TPM  +  TALDT 

Where:  OT  =  Operating  time 

ST  =  Standby  time 

TCM  =  Total  corrective  maintenance  time 

TPM  =  Total  preventive  maintenance  time 
TALDT  =  Total  administrative  and  "logistics  downtime 

o.  Operating  Time  (OT) :  The  period  of  time  that  the 
system  is  powered  and  capable  of  performing  a  mission-essential 
function. 

p.  Operational  Mission  Failure  (OMF) :  A  failure  that 
reduces  the  capability  of  the  system  to  a  point  where  one  (or 
more)  mission  essential  function(s)  cannot  be  performed. 
Measures  are  as  follows:- 

(1)  Mean  time  between  OMF  (MTBOMF) , 

(2)  Mean  miles  between  OMF  (MMBOMF) #  and 

(3)  Mean  rounds  between  OMF  (MRBOMF) . 

q.  Organizational  Level  Maintenance  (OLM) :  Maintenance 
that  is  authorized  for,  performed  by,  and  the  responsibility  of 
a  using  organization  on  its  own  equipment.  Measures  are  as 
follows: 

(1)  MTTR  at  OLM,  and 

(2)  MaxTTR  at  OLM. 

r.  Preventive  Maintenance  (PM):  The  actions  performed  to 
retain  an  item  in  a  specified  condition  by  systematic 
inspection,  detection,  and  prevention  of  incipient  failures. 

The  measure  is  Total  Preventive  Maintenance  (TPM)  time. 

s.  Percent  of  Correct  Detection:  Percent  of  all  faults  or 
failures  that  the  BIT  system  detects. 

t.  Scheduled  Maintenance:  Maintenance  that  is  performed 
on  a  regular  basis  over  the  life  of  a  system  in  order  to 
maintain  its  ability  to  perform  mission  essential  functions. 
This  maintenance  consists  of  programmed  services  and/or 
replacements  performed  at  intervals  defined  by  calender  time  or 
usage  (i.e.,  miles,  hours,  rounds...). 
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U.  Standby  Time  (ST):  The  period  of  time  that  the  system 
is  presumed  operationally  ready  for  use,  but  does  not  have 
power  applied,  is  not  being  operationally  employed,  and  no  PM 
or  CM  is  being  performed. 

v-  Time  To  Repair:  A  representative  time  of  the  effort 
expended  on  corrective  maintenance.  Measures  are  as  follows: 

(1)  Mean  time  to  repair  (MTTR),  and 

(2)  Maximum  time  to  repair  (MaxTTR) . 

w-  Unscheduled  Maintenance:  Maintenance  that  was  not 
programmed,  but  is  required  to  restore  the  item  to  partial  or 
full  mission  capability. 
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AIR  FORCE  TERMS  AND  DEFINITIONS 


1.  Purpose.  This  Appendix  provides  the  RAM  terms  and 
definitions  which  are  most  relevant  to  this  Annex  and  used 
within  the  Air  Force  in  conducting  and  reporting  OT&E 
activity.  They  have  been  extracted  from  AFR  55-43,  AFP  57-9, 
and  DcD  3235. l-H  (Test  and  Evaluation  of  System  Reliability, 
Availability,  and  Maintainability).  Tfyey  are  included  in  the 
Memorandum  of  Agreement  so  as  to  assist  other  services  in 
understanding  RAM  terms  as  used  by  the  Air  Force.  Terms  used 
by  other  services  are  included  in  Appendices  1,  2,  and  3. 

2.  Definitions 


a.  Break  Rate:  The  percent  of  time  an  aircraft  will 
return  from  an  assigned  mission  with  one  or  more  previously 
working  systems  or  subsystems  on  the  Mission-Essential 
Subsystem  List  (MESL)  inoperable  (code  3  including  ground  and 
air  aborts).  Repairs  must  be  made  before  the  aircraft  can 
perform  a  subsequent  "like-type"  mission. 

b.  Fix  Rate:  The  percent  of  aircraft,  which  return  “code 
3"  from  an  assigned  mission,  that  mufit  be  repaired  in  a 
specified  number  of  clock-hours,  i.e.,  70  percent  in  4  hours. 
Fix  rate  is  similar  to  mean  downtime.  The  time  requirement  for 
fix  rate  includes  direct  maintenance  time  and  downtime 
associated  with  maintenance  policy  and  administrative  and 
logistics  delays. 

c.  Maintainability:  The  ability  of  an  item  to  be  retained 
in  or  restored  to  specified  conditions  when  maintenance  is 
performed  by  personnel  having  specified  skill  levels,  using 
prescribed  procedures  and  resources,  at  each  prescribed  level 
of  maintenance  and  repair. 

d .  Maintenance  Man-Hours/Operating  Hour  (MMH/OH) :  The 
number  of  base-level,  direct  maintenance  man-hours  required  to 
support  a  system  divided  by  the  number  of  operating  hours 
during  the  period.  Where  aircraft,  ships,  and  vans  are 
involved,  maintenance  man-hours/flying  hour  (MMH/FH),  main¬ 
tenance  man-hours/sortie  (MMH/S),  or  some  similar  requirement 
may  be  used. 

e.  Maximum  Time  To  Repair  (MaxTTR) :  The  time  within  which 
a  specified  percentage  of  all  corrective  maintenance  tasks  must 
be  completed.  For  example,  90  percent  of  all  corrective 
maintenance  actions  must  be  completed  within  two  hours. 
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f.  Mean  Repair  Time  (MRT):  The  average  on-  or  off- 
equipment  corrective  maintenance  time  in  an  operational 
environment.  MRT  starts  when  the  technician  arrives  at  the 
system  or  equipment  for  on-equipment  at  the  system  level,  and 
off-equipment  at  the  assembly,  subassembly,  module,  or  circuit 
card  assembly  at  the  off-equipment  repair  location.  The  time 
includes  all  maintenance  actions  required  to  correct  the 
malfunction,  including  preparing  for  test,  troubleshooting, 
removing  and  replacing  components,  repairing,  adjusting,  and 
functional  check.  MRT  does  not  include  maintenance  or  supply 
delays.  MRT  is  similar  to  MTTR,  but  is  referred  to  as  MRT  when 
used  as  an  operational  term  to  avoid  confusion  with  the 
contractual  term  of  MTTR. 

g.  Mean  Downtime  (MPT):-  The  average  elapsed  clock-time 
between  loss  of  mission-capable  status  and  restoration  of  the 
system  to  mission-capable  status.  This  downtime  includes 
maintenance  and  supply  response,  administrative  delays,  and 
actual  on-equipment  repair.  In  addition  to  the  inherent  repair 
and  maintainability  characteristics,  mean  downtime  is  affected 
by  technical  order  availability  and  adequacy,  support  equipment 
capability  and  availability,  supply  levels,  and  manning.  Thus, 
MDT  is  not  the  same  as  the  contractual  term  mean  time  to  repair 
(MTTR) . 

h.  Mean  Time  Between  Critical  Failures  (MTBCF) :  The 
average  time  between  failure  of  essential  system  functions. 

For  ground  electronic  systems,  MTBCF  is  equal  to  the  total 
equipment  operating  time  in  hours  divided  by  the  number  of 
failures  of  essential  systems  required  for  completion  of  the 
assigned  mission.  MTBCF  includes  both  hardware  and  software 
failures . 

i.  Mean  Time  Between  Maintenance  Events  (MTBME)  :■  The 
average  time  between  on-equipment,  corrective  events  including 
inherent,,  induced,  no-defect,  and  preventive  maintenance 
actions.  It  is  computed  by  dividing  the  total  number  of  life 
units  (for  example,  operating  hours,  flight  hours,  rounds)  by 
the  total  number  of  maintenance  (base  level)  events  for  a 
specific  period  of  time.  A  maintenance  event  is  composed  of 
one  or  more  maintenance  actions. 

j.  Mean  Time  Between  Removal  (MTBR)  :•  A  measure  of  the 
system  reliability  parameter  related  to  demand  for  logistic 
support.  The  total  number  of  system  life  units  divided  by  the 
total  number  of  items  removed  from  that  system  during  a  stated 
period  of  time.  This  term  is  defined  to  exclude  removals 
performed  to  facilitate  other  maintenance  and  removals  for  time 
compliance  technical  orders  (TCTOs). 

k.  Mission-Capable  (MC)  Rate:  The  percent  of  possessed 
time  that  a  weapons  system  is  capable  of  performing  any  of  its 
assigned  missions.  The  MC  rate  is  the  sum  of  the  full  mission- 
capable  (FMC)  and  partial  mission-capable  (PMC)  rates. 
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l.  Operational  Availability:  The  ability  to  commit  a 
system  or  subsystem  to  operational  use  when  needed,  typically 
stated  in  terms  of  Aq,  mission-capable  rate,  sortie  generation 

rate,  or  uptime  ratio.  For  systems  with  a  defined  mission 
duration,  it  does  not  indicate  the  capability  to  complete  a 
mission  once  the  mission  begins. 

m.  Percent  BIT  Can-Not-Duplicate  (CND):  A  BIT  CND  is  an 
on-equipment,  BIT  indication  or  a  malfunction  that  cannot  be 
confirmed  by  subsequent  troubleshooting  by  maintenance 
personnel.  It  is  computed  with  the  following  formula: 

Number  of  BIT  CND 

%  CND  - - X  100 

Total  number  of  BIT  indications* 

•Excludes  false  alarms  that  do  not  generate  maintenance 
actions. 


n.  Percent  BIT  False  Alarm  (FA):  A  BIT  FA  is  an 
indication  of  a  failure  that  is  not  accompanied  by  system 
degradation  or  failure  and,  in  the  opinion  of  the  operator, 
does  not  require  any  maintenance  action.  It  is  computed  by  the 
following  formula: 

Number  of  BIT  indications  not 
resulting  in  maintenance  actions 

%  fa  - - X  100 

Total  number  of  BIT  indications 

o.  Percent  BIT  Fault  Detection  (FD):  Measures  instances 
where  a  confirmed  failure  is  a  condition  when  equipment 
performance  (including  BIT  performance)  is  less  than  that 
required  to  perform  a  satisfactory  mission,  and  corrective 
action  is  required  to  restore  equipment  performance.  The 
formula  below  assumes  that  a  requirement  exists  for  loo-percent 
diagnostics  capability. 

Number  of  confirmed  failures 
detected  by  BIT 

%  BIT  FD  - - X  100 

Number  of  confirmed  failures 
detected  via  all  methods 

p.  Percent  Fault  Isolation  (FI):-  It  is  just  as  opera¬ 
tionally  valuable  for  BIT  to  fault-isolate  an  aircrew-reported 
fault,  or  manually  detected  fault,  as  it  is  for  BIT  to  fault- 
isolate  BIT-detected  faults.  Effective  isolation  means  that 
the  fault  is  unambiguously  isolated  to  a  single-item  node 
(driver,  receiver,  connector,  wire)  or  to  a  specified  maximum 
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number  of  items  (an  ambuiguity  group  of  x  items).  The  formula 
below  defines  the  percent  of  FI. 


Number  of  fault  isolations  in  which  BIT 
effectively  contributed 


Number  of  confirmed  failures  detected  via 
all  methods 

q.  Percent  Retest  OK  ( RTOK )  :■  Defined  by  the  formula  below 
as  follows: 

Number  of  units  (LRU,  SRU)  that  RTOK  at 
a  higher  maintenance  level 

%  RTOK  - - -  100 

Number  of  units  removed  as  a  result  of 
BIT 

r-  Uptime  Ratio  (UTR) :  The  percentage  of  time  that 
operational  equipment  is  able  to  satisfy  mission  demands.  UTR 
is  similar  to  MC,  except  that  system  status  depends  on  current 
use  of  the  system,  as  well  as  the  designated  operational 
capability  (DOC) .  For  example,  a  system  with  several  DOC 
missions  can  be  MC  if  at  least  one  of  those  missions  can  be 
accomplished.  However,  if  an  immediate  need  exists  for  a 
mission  capability  that  is  "down",  the  overall  system  is 
considered  to  be  "down." 

s-  Weapon  System  Reliability  (WSR) :  The  probability  that 
a  system  will  complete  a  specified  mission  given  that  the 
system  was  initially  capable  of  doing  so. 
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